During the catalytic phase, we are developing an extensive evaluation program to assess the effectiveness of overall education as well as the diversity and inclusion programs using tools developed through NSF’s Alliance for Graduation Education and the Professoriate (AGEP) program. The Student Assessment of their Learning Gains (SALG) instrument41 is being used to evaluate the AI2EAR Fellow Program. Formative and summative assessments are being conducted under the leadership of Drs. Long and Donaldson, including assessment of design, development, and implementation of surveys/workshops; assessment of mentoring, retention and engagement activities for URM students in the education pipeline; review of data reflecting the number of URM faculty and students participating in AI2EAR; and assessment of effective incentives and engagement programs that lead to diverse and international research, mentoring, and training opportunities.

​

Overall progress towards a fully integrated network-of-network model is being assessed by an independent evaluator, Dr. Shaun Kellogg, at the Friday Institute for Educational Innovation. Thorough evaluations of the creation and implementation of workshops, student exchanges, and website creation, as well as of formative feedback on and documentation of the project team process are helping to ensure that the project activities are completed and project goals achieved. The detailed plan as it relates to process and outcomes evaluation is described below.

​

Evaluation of the Project Process:

In order to gauge the extent that project components have been implemented as intended, Dr. Kellogg will attend the kick-off meeting as well as each of the 3 annual workshops and provide assistance to the project team in the development and administration of participant surveys for workshops, student-student exchanges, and formally scheduled meetings. He will use the survey data to provide feedback to the project PIs about how well the project process is working. Dr. Kellogg will also provide feedback on the website development and collection of site analytics, including recommendations for improving user engagement. Dr. Kellogg will report to members of the EC on a quarterly basis.

​

Evaluation of Project Outcomes:

Dr. Kellogg guides the development of an activities-based logic model that defines specific activities, establishes quantifiable metrics, and specifies desired short- and long-term outcomes. Dr. Kellogg is assisting in the creation of appropriate instruments and protocols for the collection of data associated with short- and long-term outcomes and conducting external review analyses. As one of the primary goals of this project is to facilitate and grow a “Network-of-Networks,” social network analysis will play an important role in determining the extent to which implementation efforts were successful in developing connections between members of various networks, including the quality and quantity of their engagement. It is anticipated that data for network analyses will come from the following sources:

​

1. Questionnaires

   • ​​Questionnaires will be used to solicit details from participants about the quality and quantity of involvement (e.g. communication, collaboration, advice-seeking) with members of other networks.

2. Website Analytics

   • ​Trace data from website log files including user contributions (e.g. discussion posts, shared files, etc) and members’ interactions with and across networks will be used to gauge the extent to which the website has served as a useful medium for connecting members of each network.​

3. Research Publications

   • ​Finally, bibliometric networks constructed based on co-citation or co-authorship relations will be used to gauge the success of the initiative in facilitating research partnerships among network members.​

​

Data from these three sources will be used to calculate network-of-networks statistics and graphs for assessing the development over time. The evaluation of both the process and outcomes and the evolution of AI2EAR will help to ensure that we meet the goals of the project in the time allotted for the grant.