Meta-Modelling for Complex Engineered Networks

PIs: Violet R. Syrotiuk and Charles J. Colbourn

NSF NeTS Award Number: 1421058

Impact of the Project

Large-scale engineered networks, such as the Internet, the power grid, and transportation networks, play a critical role in our daily lives. In the future, the role of such networks is expected to increase in scale and scope, yet our understanding of them is limited. This is because they have evolved into complex systems with behaviours and characteristics that are beyond the characterizations and predictions possible from traditional approaches to modelling, analysis, and design.

This project has tackled the development of scientific methods for planning and assessing experiments in order to improve our understanding of complex engineered networks. The major research goals of the project were three-fold.

1. The theory of locating arrays for automatic, objective, and efficient planning of experiments.
2. The application of locating arrays for model development from experimentation in simulated and physical networks.
3. The assessment of the experimental results, i.e., the verification, validation, and robustness of the models developed.

The impact on the base of knowledge, theory, and research of each project goal follows.

In order to address some of the limitations of traditional approaches, this project introduced the theory of locating arrays for automatic, objective, and efficient planning of experiments using methods of combinatorial design. A locating array is a new scientific method for planning an experiment, i.e., for constructing the experimental design itself.

Locating arrays are formulated to focus on identification rather than on measurement. Consequently, they exhibit logarithmic growth in the number of factors. This makes practical the consideration of an order of magnitude more factors in experimentation. As a result, the use of locating arrays is novel -- they have transformed experimentation in huge factor spaces such as those found in complex engineered networks.

The effective construction of locating arrays is necessary in order to support experimentation. A framework was established for obtaining asymptotic existence results, generalizing a sequence of improvements on the best known sizes of covering arrays to locating arrays. A breakthrough on using covering perfect hash families for locating and detecting arrays (which support stronger location properties) yielded substantial improvements on the asymptotic results.

The asymptotic methods each underlie efficient algorithms for the construction of locating arrays, using column resampling techniques and conditional expectation methods. The algorithmic methods were extended to treat a `separation' among the tests, to enhance the robustness of the analysis. The novel approach using covering perfect hash families also supports algorithmic extensions for much larger testing environments.

Computational methods are limited by the combinatorial explosion in the number of interactions as the number of factors increases. Recursive methods can be used to construct locating arrays for many factors from locating arrays for few. Powerful recursive methods have been developed for locating arrays of strength two, as well as a different recursive method based on hash families.

A major challenge is the analysis of data sets from the application of locating arrays in experimentation, because locating arrays may be highly unbalanced. This makes existing software tools inadequate. We have therefore developed new scientific methods for assessing experiments. We have been inspired by a "heavy hitters'' approach, and an l1-recovery technique, used to recover sparse signals from compressive sensing matrices. Compressive sensing matrices used are typically constructed using probabilistic algorithms, with the matrix satisfying certain properties with high probability. We instead use deterministic methods to guarantee that our locating arrays satisfy the locating property.

We adapted the orthogonal matching pursuit (OMP) framework of compressive sensing. The algorithm is iterative and in each iteration it selects the most significant term to add to the model of the experimental response. The selection is facilitated by the structure of the locating array. The model is then refit using least squares and the residuals are recalculated. These steps repeat until a stopping criterion is met. To counter the problem in the OMP framework that can only add terms to a model, a branch-and-bound component is integrated to consider alternative options for the selections made using OMP. This takes into account noise in measurements that often occur in real systems and also measurements that are outliers.

Assessing the experimental results, i.e., the verification, validation, and robustness of the developed models is critical to the accuracy of the model. For real engineered networks and systems, the true model is unknown. Hence the robustness of developed models has been evaluated using synthetic data generated from known models. Accurate assessment on synthetic data improves confidence in the behaviours and characterizations of real network systems produced.

Complex engineered networks have shaped modern society and form part of our critical infrastructure. The ability to plan and assess experiments effectively in very large factor spaces has broad impact to science and engineering. Locating arrays are a shift rather than a step forward in the field, transforming the frontiers and innovating for society.

The modelling approach based on locating arrays works to address some of the grand challenges in networks for new data-driven mathematical models. Such models facilitate quantitative evaluation and contribute to our understanding of complex engineered networks.

At the conclusion of our project, we have developed a new methodology for scientific evaluation of complex engineered systems, through the development of rigorous scientific methods for planning and assessing experiments. We have impacted the base of knowledge and theory of locating arrays in combinatorics, and the design, analysis, and validation of experiments. Our research has led to results driven by empirical data that are statistically illuminating and reproducible in the field of complex engineered networks.

The Major Goals of the Project

The major research goals of the "Meta-Modelling for Complex Engineered Networks" project include:

1. The theory of locating arrays for automatic, objective, and efficient planning of experiments using methods of combinatorial design.
2. The application of locating arrays for meta-model development by experimentation in both simulated and physical wireless networks using a "heavy hitters" approach and non-parametric statistical techniques.
3. The assessment of the experiments, i.e., the verification, validation, and robustness of the developed meta-models.

The Major Accomplishments of the Project

We summarize accomplishments under the three major research goals of the project.

Goal 1: The Theory of Locating Arrays

The effective construction of locating arrays (LAs) is necessary in order to support experimentation. Constructions fall into four categories: Exact methods, asymptotic results, computational methods, and recursive methods.

Exact Methods: The determination of the minimum size of an LA in general is challenging. Using techniques from graph decomposition, linear optimization, and combinatorial enumeration, we have completely determined the minimum size of an array to locate one main effect.

Asymptotic Results: To a large degree, advances on LAs necessitate techniques for the simpler covering arrays that can be suitably generalized. Probabilistic techniques that lead to asymptotic bounds lend themselves to such generalization well, so we have focussed on asymptotic existence results for covering arrays. Using group action on the symbols of the array, we have improved on the best previous asymptotic existence result for covering arrays. Subsequently, we extended these techniques to treat the relaxation in which most, but perhaps not all, interactions are covered; we obtained tight bounds for the array sizes.

We established that using a "permutation vector" representation of covering arrays underlies a further dramatic improvement in the asymptotic bounds for covering arrays. Two general strategies, one based on the Stein-Lovasz-Johnson theorem, and one based on the Lovasz Local Lemma (LLL), both lead to improvements, but the LLL strategy is asymptotically more accurate. These techniques apply to LAs directly.

Part of this effort shows that the methods to produce the bounds yield construction algorithms that are not only efficient in theory, but are also practical for experiments involving thousands of factors.

Computational Methods: Two-stage algorithms for construction lead to effective improvements on the known sizes of covering arrays. The key to success in the two-stage methods is that a first stage can cover almost all interactions in a random way, allowing a second stage to use computationally more intensive methods such as density and graph colouring in order to reduce the array size. We are using a similar approach to exploit the permutation vector representation; our work suggests that for strengths at least three, we can explicitly construct covering arrays that are the best available across a wide range of parameters.

Although the novel methods for covering arrays provide a solid foundation, our ultimate interest is in the effective construction of LAs. We continue to explore extensions of the two-stage covering array methods to LAs. Patterned on the advances for covering arrays, our ongoing work to construct LAs is employs the techniques of column resampling, conditional expectation methods, and post-optimization.

Recursive Methods: No matter how effective the asymptotic results and computational methods become, the combinatorial explosion in the number of interactions as the number of factors increases ensures that simply checking that an array is locating becomes infeasible. To circumvent this, recursive methods can be used to make LAs for many factors from LAs for few. We have developed powerful recursive methods for LAs of strength two, patterned on known methods for covering arrays. We have also developed a number of improvements in hash family asymptotics and constructions, and explored their use in recursive constructions for LAs.

Parameters of Locating Arrays: We defined two new parameters of locating arrays that are useful in experimental settings. A separation of w in that ensures that fewer than w missed measurements or outliers cannot result in an interaction’s impact being lost. Corroboration enables the fusion of values while maintaining the presence of witnesses. Constructions having various values for the separation and corroboration use error correcting codes and separating hash families.

General Parameters: Our effort has focussed on the construction of (1,2)-LAs because of our analysis method exploits these parameters. We have a new construction for more general (d,t)-LAs, which have the property that every set of d t-way interactions appears in a unique set of tests, based on a new randomized computational search algorithmic framework. It produces many fewer tests than other known general methods. This construction may underlie a new recovery algorithm, that may allow weakening the heavy-hitters assumption.

Goal 2: The Application of Locating Arrays for Meta-Model Development

Experimentation in Physical Wireless Networks: An audio-conferencing experiment was designed the w-iLab.t wireless network in Ghent, Belgium, with one speaker broadcasting and an audience listening. A total of 24 factors were selected affecting the kernel's IP and UDP stacks, the Wi-Fi card driver, the audio codec, and a source of interference. Each factor had from 2 to 5 levels, with many being categorical. Two responses were collected: The mean opinion score (MOS), computed from delay, jitter, and packet loss, and radio frequency (RF) exposure. A full factorial design for this set of factors has trillions of runs while an LA has only 109 runs.

Experimentation in Simulation: We conducted an experiment in ns-2, a simulated wireless network with 75 controllable factors spanning the TCP/IP protocol stack, each ranging from 2 to 10 levels, in which we measured the average TCP throughput. The full-factorial design for this factor space has over 5 x 10^{43} runs! In contrast, the LA has only 421 runs.

With Prof. Famaey at the University of Antwerp, we study 16 factors to determine the effect on delay, aggregate throughput, energy consumption, and packet loss in a simulated IEEE 802.11ah network. IEEE 802.11ah, or Wi-Fi HaLow, is a new Wi-Fi draft for sub-1GHz communications aiming to address the major challenges of the Internet of Things (IoT). Two new issues arise from this work that impacted the construction of the LA and its use in analysis. The first is that IEEE 802.11ah has a number of factors each having a very large number of values, while the second concerns constraints on the parameter values. We are analyzing the results of IEEE 802.11ah simulations, collected from LAs constructed using a new parameter grouping technique with some simple, feasible constraints among parameter values.

With Prof. Tinnirello at the University of Palermo, we explore LoRa, which enables large scale deployment of low power wide area networks (LPWANs). We have chosen a simulation of a LoRa water metering scenario based on a physical IoT system in actual use to begin an investigation of explainable AI (XAI) techniques; our LA-based analysis technique appears related. We are interested to see which factors of the LoRa scenario that XAI identifies as significant on performance metrics such as the data extraction rate. In particular, we are interested how much training data is required for XAI to accomplish the identification. We are exploring whether a more structured approach to training, such as one based on LAs, may improve XAI and/or reduce the amount of training data that needs to be used.

Goal 3: The Assessment of the Experiments and Robustness of the Developed Meta-Models

A major challenge encountered in the analysis of the data sets collected through experimentation where an LA is used as the experimental design. Because LAs are often highly unbalanced, existing software tools for analysis (such as JMP and minitab) are inadequate. Therefore we developed a new scientific methods for assessing experiments based on LAs.

Our initial method used a parametric approach. If first grouped main effects and two-way interactions according to how many times each is measured in the LA, to account for imbalance. The Wilcoxon rank sum and Mann-Whitney U-test were used to obtain the most significant term in each group. Then from these candidates, the most significant main effects or two-way interaction overall was selected using the Akaike information criterion.

We then developed a non-parametric approach inspired by a "heavy-hitters" and an l1-recovery technique used to recover sparse signals from compressive-sensing matrices. We adapted the orthogonal matching pursuit (OMP) framework of compressive sensing. Our algorithm is iterative and in each iteration it selects the most significant term (factor or two-way interaction) to add to the model of the experimental response. This is facilitated by structure of the array because it is a (1,2)-LA and is specifically designed to identify one, main effect or two-way interaction. The model is then refit using least squares and the residuals are recalculated. These steps repeat until a stopping criterion is met. While the resulting model is not intended to be predictive, it is often a good predictor.

To counter the problem that OMP can only add terms to a model, the algorithm constructs many models simultaneously to consider alternative options for the selections made using OMP. This takes into account noise in measurements, and measurements that are outliers, that occur often in real systems. First, we developed alternative models using a depth-first search (DFS) with back-jumping. To improve the efficiency of the algorithm we replaced the DFS with breadth-first search (BFS).

Because the true model of response in the simulated or testbed system is unknown we generated synthetic data from known models where the coefficients were varied. If the coefficients of the known models satisfy the heavy-hitters assumption, the model recovered is exact. As they deviate from the assumption, the recovery is still quite good. When noise is added to the data, recovery works well even for remarkably high noise. However, if the known model has a very large number of terms then the terms with lower coefficients become unrecoverable. Similarly, the terms with very small coefficients become unrecoverable because they are indistinguishable from noise. These results confirm our intuition.

In order for LAs and their analysis to be accepted as a novel design by the design-of-experiments (DoE) and statistics communities, its merits have to be argued in terms those communities understand. We reached out to Prof. Stufken, a statistician at the University of North Carolina, Greensboro, who also has expertise in combinatorics, to focus on the analysis of the experimental results. Our analysis algorithm uses a compressive sensing matrix to identify the term in each iteration of the algorithm that was closest to the residuals. We have compared to using more conventional coding matrix and more conventional selection methods (such as the Dantzig selector method), both more commonly used in DoE techniques; the results are very promising. Further, it shows that the LA can be used as an experimental design and analyzed by conventional methods if those methods are preferred.

When possible, models developed from data collected from experiments based on LAs have been validated with analytic models published in the literature. We have also compared to models developed with the JMP statistical software from full-factorial experiments on a subset of factors present in our model.

Opportunities for Training and Professional Development

The "Meta-Modelling for Complex Engineered Networks" project has provided many opportunities for training and professional development. All students received training in research through one-on-one meetings with their supervisor, i.e., one or both of Professor Syrotiuk or Professor Colbourn.

Several students had the opportunity to collaborate with peers internationally or nationally. Collaborations allow students to experience a different academic culture, and see that their research is contributing to that of a global research community. It is highly motivating for a student to be involved in a collaboration because his or her work (or lack of work) impacts the entire group, not just him or herself; hence it can increase the productivity of the student.

Several students also took advantage of opportunities for professional development. These develop and improve the skills of students.

International Collaborations

1. Ph.D. student C. Randy Compton had the opportunity to travel to Ghent, Belgium with the PIs. This travel was supported by an award supporting collaboration between NSF GENI and EU FIRE researchers. Randy collaborated with Prof. Ingrid Moerman's of Ghent University - imec, and members of her group. Randy worked with peer Michael Mehari on configuring and running locating-array based experiments on the w-iLab.t wireless network testbed; this work is related to research goals (2) and (3).
2. Ryan Dougherty, a Ph.D. student, collaborated with Professor Daniel Horsley at Monash University in Australia, on the topic of hash families. He has also collaborated with Drs. Dimitris Simos and Kristoffer Klein at SBA Research in Austria on covering arrays and hash families. Ryan's work was related to research goal (1).
3. Nathaniel Flick, an undergraduate, chose an ambitious project for his senior honours thesis. He implemented the meta-MAC protocol on a wireless testbed at the University of Palermo, and conducted numerous experiments. Nathan collaborated with Domenico Garlisi, a member of Professor Ilenia Tinnirello's team; his work was related to research goal (2).
4. Daniel Kulenkamp was an undergraduate student whose senior honours thesis extended the work of Matt Mellott related to medium access control in wireless networks on the w-iLab.t testbed. With the PIs he visited Professor Ilenia Tinnirello and her group at the University of Palermo. That visit was supported by an international travel grant from the GENI Project Office and Boston University, under NSF collaborative agreement CNS-1536090. Daniel has also collaborated with Professor Johann Marquez-Barja, and his Ph.D. student Pedro Heleno Isolani, at the University of Antwerp - imec. Daniel is now an M.S. student; his work is related to research goal (2).
5. Matthew Mellott was an M.S. student who collaborated with Professor Ilenia Tinnirello and her group at the University of Palermo, on medium access control schemes for wireless networks. This involved the implementation of a protocol and experimentation on w-iLab.t in Ghent and was related to research goal (2).
6. Kaushik Sarkar, a Ph.D. student supported by the grant, collaborated with Professors Ugo Vaccaro and Annalisa De Bonis of the University of Salerno. He is also collaborated with Professors Bingli Fan and Tao Fang at Beijing Jiaotong University on computing asymptotic bounds for locating arrays. Kaushik's work was related to research goal (1).
7. Stephen Seidel started work on the project an undergraduate student and earned his M.S. in the 4+1 program. Stephen wrote code to construct locating arrays, to perform the analysis of data collected from experimentation, and conducted experiments to validate the models constructed. He collaborated with Professor Ingrid Moerman's group at Ghent University - imec, and also Professor Jeroen Famaey and his Ph.D. student at the University of Antwerp - imec. Stephen's work was related to all three research goals.

National Collaborations

1. Yasmeen S. Akhtar collaborated with Professor John Stufken of the University of North Carolina, Greensboro, and his Ph.D. student Fan Zhang, on research direction (3).

International Professional Development Opportunities

1. Yasmeen S. Akhtar participated in two conferences held back-to-back in Vancouver, B.C., Canada. She presented research at The 7th Biennial Canadian Discrete and Algorithmic Mathematics Conference (CanaDAM), and also at the 14th International Conference on Finite Fields and their Applications (FQ14). She obtained some support from each conference for her travel.
2. Daniel Kulenkamp won a travel award to participate in MobiHoc'19, held in Catania, Sicily, Italy. It was his first opportunity to attend a premier international symposium dedicated to addressing challenges in dynamic networks and computing.
3. Erin Lanus had the opportunity to present her research at the Conference on Combinatorics and its Applications in Singapore, supported by a travel grant. Erin also presented her Partitioned Search with Column Resampling (PSCR) algorithm at the Workshop on Software Testing, Verification and Validation (ICSTW) in Xi'an, China. Her travel was supported by her scholarship.
4. Muhilan Rammamoorthy, a Ph.D. student, won a travel award to attend the third Fed4Fire-GENI Camp held at Ghent University - imec, in Ghent, Belgium. There, he gained hands-on experience with running networking and distributed systems experiments using resources and tools from the U.S. GENI and the European FIRE research infrastructures. Specifically, the camp included presentations and hands-on tutorials on wireless and sensor networks, internet of things (IoT), smart cities, big data and clouds, and software defined networking. Muhilan's research interests are in the areas of IoT and smart cities. He is examining problems of waste collection management modelled as a variant of the capacitated arc routing problem (CARP) in smart cities. To date, his work has involved experimentation in simulation.
5. M.S. student Felipe Stall Rechia won a travel award to attend and participate in the Fed4FIRE-GENI Research Experiment Summit held at the University of Ghent - imec, in Ghent, Belgium. As a participant, Felipe learned to use resources and tools available in the Fed4FIRE and the GENI infrastructure, and participated in running experiments using tools such as OMF, OML, and LabWiki, and tutorials on topics including OpenFlow, wireless and sensor networks, and smart city sensors in Smartsantander. These activities prepared Felipe to conduct experimentation on network testbeds related to research goal (2) and contributed to the research direction pursued in his thesis.

National Professional Development Opportunities

1. Yasmeen S. Akhtar presented a paper on Locating Arrays and their Application as Screening Designs, at the Design and Analysis of Experiments Conference (DAE), in Knoxville, Tennessee, October 2019.
2. Randy Compton presented work on design and analysis of experiments based on locating arrays at CNERT'16 in San Francisco, CA.
3. Nathaniel Flick presented his work on the meta-MAC protocol at CNERT'16 in San Francisco, CA.
4. Stephen Seidel presented his results on analysis at CNERT'18 in Honolulu, HI, and also at SpringSim'19 in Tucson, AZ, on validation of the analysis method.

Dissemination of Results

Over the life of the "Meta-Modelling for Complex Engineered Networks" project the results of have been disseminated in 17 (refereed) journal publications, 2 (refereed) book chapters, and 13 (refereed) conference proceedings.

In addition, a total of 4 Ph.D. dissertations, 5 M.S. theses, and 3 undergraduate honours theses have been defended. Several Ph.D., M.S., and undergraduate research projects are ongoing.

The results of the project were also disseminated over the years through many invited talks and seminars.

Invited Talks:

1. C.J. Colbourn, Fault Location and Resolvable Set Systems, Algebraic Combinatorics and Applications conference, Houghton, Michigan, August 2015.
2. V. R. Syrotiuk, Making WiFi Work in Multi-Hop Topologies: Automatic Negotiation and Allocation of Airtime, Annual Meeting of the Sociedad Mexicana de Ciencias de la Computacion (SMCC'15), Ensenada, Baja California, Mexico, October 2015.
3. C. J. Colbourn, The Construction of Locating Arrays, The 29th Midwestern Conference on Combinatorics and Combinatorial Computing, Charleston SC, October 2015.
4. C. J. Colbourn, Disjoint Spread Systems and Fault Location, Auburn Conference on Designs, Graphs, and Codes, Auburn AL, January 2016.
5. C. J. Colbourn, Locating Arrays and Disjoint Partitions, Software Developers Association, Phoenix AZ, March 2016.
6. C. J. Colbourn, Coverage, Location, Detection, and Measurement, IWCT 2016 : 5th International Workshop on Combinatorial Testing, Chicago IL, April 2016.
7. V. R. Syrotiuk, Screening Experiments: Do Interactions Matter?, The 12th IEEE International Workshop on Wireless Networks: Measurements and Experimentation (WiNMeE'16), Tempe AZ, May 2016.
8. C. J. Colbourn, Coverage, Location, Detection, and Measurement, Discrete Math Days, Ottawa, Canada, May 2016.
9. C. J. Colbourn, Covering Arrays: Asymptotics and Constructive Algorithms, Nordic Combinatorial Conference, Levi, Finland, June 2016.
10. C. J. Colbourn, Disjoint Spread Systems and Fault Location, National Conference on Combinatorial Designs, Zhejiang University, Hangzhou, China, July 2016.
11. C. J. Colbourn, Asymptotic and Constructive Bounds for Covering Arrays, Real and Complex Hadamard Matrices and Applications, Budapest, Hungary, July 2017.
12. C. J. Colbourn, Fervour with Measure: The Mathematics of Alex Rosa, RosaFest: Alex Rosa is 80, Mikulov, Czech Republic, June 2017.
13. V. R. Syrotiuk, Using Locating Arrays for Screening in a Wireless Network Testbed, RosaFest: Alex Rosa is 80, Mikulov, CZ Republic, June 2017.
14. C. J. Colbourn, Computational and Recursive Constructions of Perfect Hash Families, Seventh International Conference on Algebraic Informatics (CAI'17), Kalamata, Greece, June 2017.
15. C. J. Colbourn, Asymptotic Sizes of Covering Arrays, HyGraDe: Hypergraphs, Graphs, and Designs, Sant'Alessio Siculo, Italy, June 2017.
16. C. J. Colbourn, Covering Perfect Hash Families, Ninth Shanghai Conference on Combinatorics (9SHCC), Shanghai, China, May 2017.
17. C. J. Colbourn, Locating Interactions: Separation and Constraints, Workshop on Combinatorics and its Applications, Singapore, July 2018.
18. C. J. Colbourn, Asymptotic and Constructive Bounds for Sequence Covering Arrays,' Thirtieth Cumberland Conference on Combinatorics, Huntington WV, May 2018.
19. C. J. Colbourn, Asymptotic and Constructive Bounds for Hash Families, Zhejiang Workshop on Difference Sets, Hangzhou, China, May 2018.
20. C. J. Colbourn, Asymptotic and Constructive Bounds for Hash Families, Fifth International Conference on Combinatorial Mathematics, Monash, Clayton, Australia, December 2017.
21. C. J. Colbourn, Why Can't They Just Work Together?, Faculty of Science Interdisciplinary Lecture Series, University of Manitoba, Winnipeg, Canada, November 2017.
22. C. J. Colbourn, Locating Interactions: Separation and Constraints, Conference on Combinatorics and its Applications, Nanyang Technological University, Singapore, July 2018.
23. V. R. Syrotiuk, Locating Arrays and their Application in Screening Wireless Networks, Conference on Combinatorics and its Applications, Nanyang Technological University, Singapore, July 2018.
24. V. R. Syrotiuk, Locating Arrays: A New Experimental Design for Complex Engineered Systems, Cybersecurity Analytics and Applications session of the Military Applications Society track at the Annual INFORMS Conference. Phoenix, AZ, U.S.A., November 2018.
25. V. R. Syrotiuk, Design and Analysis of Experiments, Keynote Address, Chameleon User Meeting. Austin, TX, U.S.A., February 2019.

Seminars:

1. V. R. Syrotiuk, Making WiFi Work in Multi-Hop Topologies: Automatic Negotiation and Allocation of Airtime, ASU Math Club, Tempe, AZ, October 2015.
2. V. R. Syrotiuk, Locating Arrays and Screening Interactions, Seminar, Department of Mathematics, Beijing Jiaotong University, Beijing, China, July 2016.
3. C. J. Colbourn, Covering Arrays: Asymptotics and Constructive Algorithms, Mathematics Seminar, Beijing Jiao Tong University, Beijing, China, July 2016.
4. V. R. Syrotiuk, Making Wi-Fi Work in Multi-Hop Topologies: Automatic Negotiation and Allocation of Airtime, Fishbowl Seminar Series, Texas A\&M University, February 2017.
5. V. R. Syrotiuk, Screening Experiments: Do Interactions Matter?, Seminar, Department of Information Technology, Ghent University, Ghent, Belgium, September 2017.
6. C. J. Colbourn, Perfect Hash Families with Few Rows, Mathematics, Beijing Jiao Tong University, Beijing, China, October 2017.
7. C. J. Colbourn, Covering Perfect Hash Families: Algorithms, Mathematics, Beijing Jiao Tong University, Beijing, China, October 2017.
8. C. J. Colbourn, Asymptotic Bounds for Covering Arrays, Mathematics, Beijing Jiao Tong University, Beijing, China, October 2017.
9. V. R. Syrotiuk, Realizing Airtime Allocations in Multi-Hop Wi-Fi Networks: A Stability and Convergence Study with Testbed Evaluation, Seminar IDLab, Department of Elektronics-ICT, University of Antwerp - imec, Antwerp, Belgium, April 2019.

Publications

Book Chapters:

1. C. J. Colbourn and V. R. Syrotiuk (2019). There Must be 50 Ways to Miss a Cover. 50 Years of Combinatorics, Graph Theory, and Computing, F. Chung, R. Graham, R. Hoffman, L. Hogben, R. Mullin, and D. West, eds. CRC Press. Chapter 18, pages 319-334.
2. C. J. Colbourn and V. R. Syrotiuk. Covering Strong Separating Hash Families. To appear in Finite Fields and their Applications, J. Jedwab and P. Lisonek, eds. De Gruyter.

Journal Papers:

1. A. N. Aldaco, C. J. Colbourn, and V. R. Syrotiuk (2015). Locating Arrays: A New Experimental Design for Screening Complex Engineered Systems. SIGOPS Operating Systems Review. 49 (1), 31. DOI: 10.1145/2723872.2723878
2. C. J. Colbourn and B. Fan (2016). Locating One Pairwise Interaction: Three Recursive Constructions. Journal of Algebra Combinatorics Discrete Structures and Applications. 3 125.
3. C. J. Colbourn, B. Fan, and D. Horsley (2016). Disjoint Spread Systems and Fault Location. SIAM Journal on Discrete Mathematics. 30 2011.
4. K. Sarkar and C. J. Colbourn (2017). Upper bounds on the size of covering arrays. SIAM Journal on Discrete Mathematics. 31 1277. arXiv identifier 1603.07809
5. D. Bryant, C. J. Colbourn, D. Horsley, and I. M. Wanless (2017). Steiner triple systems with high chromatic index. SIAM Journal on Discrete Mathematics. 31 2603.
6. D. Bryant, C. J. Colbourn, D. Horsley, and P. O Cathain (2017). Compressed sensing with combinatorial designs: theory and simulations. IEEE Transactions on Information Theory. 63 4850.
7. C. J. Colbourn, E. Lanus, and K. Sarkar (2018). Asymptotic and constructive methods for covering perfect hash families and covering arrays. Designs, Codes and Cryptography. 86 907.
8. C. J. Colbourn and E. Lanus (2018). Subspace Restrictions and Affine Composition for Covering Perfect Hash Families. Art of Discrete and Applied Mathematics. 1 #P2.03.
9. C. J. Colbourn and V. R. Syrotiuk (2018). On a Combinatorial Framework for Fault Characterization. Mathematics in Computer Science. 12 (4), 429. DOI: 10.1007/s11786-018-0385-x
10. C. J. Colbourn, D. Horsley, and V. R. Syrotiuk (2018). A Hierarchical Framework for Recovery in Compressive Sensing. Discrete Applied Mathematics. 36 96.
11. K. Sarkar, C. J. Colbourn, A. De Bonis, and U. Vaccaro (2018). Partial covering arrays: Algorithms and Asymptotics. Theory of Computing Systems. 62 1470.
12. S. Maity, Y. Akhtar, R. C. Chandrasekharan, and C. J. Colbourn (2018). Improved Strength Four Covering Arrays with Three Symbols. Graphs and Combinatorics. 34 223.
13. K. Sarkar and C. J. Colbourn (2019). Two-stage algorithms for covering array construction. Journal of Combinatorial Designs. 27 475. DOI: 10.1002/jcd.21657
14. M.J. Mellott, D. Garlisi, C.J. Colbourn, V.R. Syrotiuk, and I. Tinnirello (2019). Realizing airtime allocations in multihop Wi-Fi networks: A stability and convergence study with testbed evaluation. Computer Communications. 145 273. DOI: 10.1016/j.comcom.2019.07.006
15. C. J. Colbourn, R. E. Dougherty, and D, Horsley (2019). Distributing Hash Families with Few Rows. Theoretical Computer Science 800, 31-41.
16. C. J. Colbourn and V. R. Syrotiuk. Detecting Arrays for Effects of Single Factors. To appear in Information and Computation.
17. Y. Chang, C. J. Colbourn, A. Gowty, D. Horsley, and J. Zhou. New Bounds on the Maximum Size of Sperner Partition Systems. To appear in European Journal of Combinatorics.

Conference Papers:

1. D. Garlisi, F. Giuliano, A. Lo Valvo, J. Lutz, V. R. Syrotiuk, and I. Tinnirello (2015). Making WiFi Work in Multi-Hop Topologies: Automatic Negotiation and Allocation of Airtime. Proceedings of the International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT'15), in conjunction with IEEE 35th International Conference on Distributed Computing Systems (ICDCS'15). 48. DOI: 10.1109/ICDCSW.2015.20
2. C. J. Colbourn and V. R. Syrotiuk (2016). Coverage, Location, Detection, and Measurement (Invited). 5th International Workshop on Combinatorial Testing (IWCT'16), in conjunction with IEEE International Conference on Software Testing, Verification and Validation (ICST'16).
3. I. Tinnirello, D. Garlisi, F. Giuliano, V. R. Syrotiuk, and G. Bianchi (2016). Demo: MAC Learning: Enabling Automatic Combination of Elementary Protocol Components. Proceedings of the 10th ACM International Workshop on Wireless Testbeds, Experimenal evaluation and CHaracterization (WINTECH'16), in conjunction with the 22nd ACM International Conference on Mobile Computing and Networking (MobiCom'16).
4. K. Sarkar, C. J. Colbourn, A. De Bonis, and U. Vaccaro (2016). Partial Covering Arrays: Algorithms and Asymptotics. Lecture Notes in Computer Science, International Workshop on Combinatorial Algorithms (IWOCA16), Helsinki FI. arXiv identifier 1605.02131
5. N. Flick, D. Garlisi, V. R. Syrotiuk, and I. Tinnirello (2016). Testbed Implementation of the Meta-MAC Protocol. Proceedings of the International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT'16), in conjunction with IEEE International Conference on Computer Communications (INFOCOM'16).
6. R. Compton, M. T. Mehari, C. J. Colbourn, E. De Poorter, and V. R. Syrotiuk (2016). Screening Interacting Factors in a Wireless Network Testbed Using Locating Arrays. Proceedings of the International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT'16), in conjunction with IEEE International Conference on Computer Communications (INFOCOM'16).
7. M. J. Mellott, C. J. Colbourn, V. R. Syrotiuk, and I. Tinnirello (2018). Testbed Evaluation of Optimized REACT with Multi-hop Extensions. Proceedings of the 16th International Conference on Wired/Wireless Internet Communications (WWIC'18), Boston, Massachusetts, U.S.A..
8. S. A. Seidel, K. Sarkar, C. J. Colbourn, and V. R. Syrotiuk (2018). Separating Interaction Effects using Locating and Detecting Arrays. Proceedings of the International Workshop on Combinatorial Algorithms (IWOCA'18), Singapore.
9. S. A. Seidel, M. T. Mehari, C. J. Colbourn, E. De Poorter, I. Moerman, and V. R. Syrotiuk (2018). Analysis of Large-Scale Experimental Data from Wireless Networks. Proceedings of the International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT'18), in conjunction with IEEE International Conference on Computer Communications (INFOCOM'18), Honolulu, Hawaii, U.S.A.
10. C. J. Colbourn and V. R. Syrotiuk (2019). Detecting Arrays for Main Effects. Lecture Notes in Computer Science. 11545 112. DOI: 10.1007/978-3-030-21363-3_1
11. E. Lanus, C. J. Colbourn, and D. C. Montgomery (2019). Partitioned Search with Column Resampling for Locating Array Construction. Proceedings of the 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). DOI: 10.1109/ICSTW.2019.00056
12. S. A. Seidel, C. J. Colbourn, and V. R. Syrotiuk (2019). Robustness of Recovery in Locating Array-based Screening Experiments. Proceedings of the SCS Spring Simulation Conference (SpringSim). DOI: 10.23919/SpringSim.2019.8732911
13. M. Ramamoorthy and V. R. Syrotiuk (2020). Online Re-routing for Vehicle Breakdown in Residential Waste Collection. To appear in Proceedings of the Vehicular Technology Conference (VTC-Fall).

Theses:

1. Abraham N. Aldaco-Gastelum, "A Framework for Screening Experiments and Modelling in Complex Systems," Ph.D. Dissertation, Spring 2015.
2. Nicole Ang, "Generating Mixed-Level Covering Arrays with λ = 2 and Test Prioritization," MS Thesis, Spring 2015.
3. Rushang Karia, "Covering Arrays: Generation and Post-Optimization," MS Thesis, Spring 2015.
4. Kaushik Sarkar, "Covering Arrays: Algorithms and Asymptotics," Ph.D. Dissertation, Fall 2016.
5. Nathaniel Flick, "Testbed Implementation of the Meta-MAC Protocol," Senior Honours Thesis, Spring 2016.
6. Felipe Stall Rechia, "An Evaluation of SDN Based Network Virtualization Techniques," MS Thesis, Spring 2016.
7. Matthew J. Mellott, "Smoothed Airtime Linear Tuning and Optimized REACT with Multi-hop Extensions," MS Thesis, Spring 2018.
8. Stephen A. Seidel, "Locating Arrays: Construction, Analysis, and Robustness," MS Thesis, Fall 2018.
9. Erin Lanus, "Interaction Testing, Fault Location, and Anonymous Attribute-Based Authorization," PhD Dissertation, Spring 2019.
10. Ryan E. Dougherty, "Hash Families and Applications to t-Restrictions," PhD Dissertation, Spring 2019.
11. Vincent Miller, "Constructing Locating Arrays with Constraints using Constraint Satisfaction," Senior Honours Thesis, Spring 2019.
12. Daniel Kulenkamp, "Improving on 802.11: Streaming Audio and Quality of Service," Senior Honours Thesis, Fall 2019.

Code:

Stephen A. Seidel, Code to Generate Locating Arrays and Analyze Experimental Results based on Locating Arrays, https://github.com/sseidel16/v4-la-tools

Collaborations:

1. Professors Ingrid Moerman and Eli De Poorter, and their team, IDLab - Internet Technology and Data Science Lab, Ghent University - imec, Ghent, Belgium.
2. Professor Ilenia Tinnerello and her team, Department of Electrical Engineering, University of Palermo, Palermo, Italy.
3. Professors Joroen Famaey and Johaann Marquez-Barja, IDLab - Internet Technology and Data Science Lab, University of Antwerp - imec, Antwerp, Belgium.
4. Professors Yanxun Chang, Junling Zhou, Bingli Fan, and Tao Feng, Department of Mathematics, Beijing Jiaotong University, Beijing, China.
5. Professor Daniel Horsley, School of Mathematical Sciences, Monash University, Melbourne, Australia.
6. Professors Ugo Vaccaro and Annalisa De Bonis, Department of Computer Science, University of Salerno, Salerno, Italy.
7. Professor John Stufken, Department of Mathematics and Statistics, University of North Carolina, Greensboro, U.S.A.
8. Drs. Dimitris Simos and Kristoffer Klein, SBA Research, Vienna University of Technology, Vienna, Austria.