10.46243/jst.2021.v6.i06.pp139-149 registered
Using Benchmarking to Advance Research: A Challenge to Software Engineering
Resolves to https://www.jst.org.in/index.php/pub/article/view/532
Held by Longman Publishers (India) · prefix 10.46243 live · DOI address https://doi.org/10.46243/jst.2021.v6.i06.pp139-149
Registered 29 Sep 2026 via crossref · record version 2 · last change 30 Sep 2026, 12:00 AM · record sha256 2cf2633011cd7cc0…
Resolve ⬇ Record (JSON) ⬇ Kernel Metadata Declaration (XML) Compare with Crossref Cite (APA · BibTeX · RIS · CSL)
Also as schema.org JSON-LD · Turtle · RDF/XML · DataCite XML · Crossref UNIXREF · Crossref UNIXSD — the formats of the DOI content-negotiation service, from this record (which carries what).
What the DOI identifies
JournalArticle — an article in a journal · Digital · Visual · en
Using Benchmarking to Advance Research: A Challenge to Software Engineering (PrincipalTitle · en)
Published 2021
Part of Journal of Science & Technology · ISSN 2456-5660 · volume 06 · issue 06 · pages 139–149
Agents
- K Vijay Kumar (author)
- Dr.Y V R Naga Pawan (author)
- Longman Publishers (publisher)
Identifiers DOI 10.46243/jst.2021.v6.i06.pp139-149
Abstract · en
Benchmarks have been used in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. These observations have led us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. In support of this challenge, we present a case study of the reverse engineering community, where we have successfully used benchmarks to advance the state of research.
System metadata — ISO 26324:2025, Annex B · DOI Handbook 10.1
Each element by the standard's name (Annex B: reference elements, then administrative) and the Handbook's (in grey), read off the record above.
| Element | Value | In the record |
|---|---|---|
| DOI Name DOI name | 10.46243/jst.2021.v6.i06.pp139-149 | doi |
| Referent Type referentType | Creation | referent |
| Referent Sub-Type referentSubType | JournalArticle — an article in a journal | type |
| Referent Name(s) referentName(s) | Using Benchmarking to Advance Research: A Challenge to Software Engineering (PrincipalTitle, en) | titles |
| Basic Metadata basicMetadata | author: K Vijay Kumar author: Dr.Y V R Naga Pawan publisher: Longman Publishers published: 2021 part of: Journal of Science & Technology · ISSN 2456-5660 · vol. 06 · no. 06 · pp. 139–149 language: en form: Digital · Visual · Language | agents, dates, container, language, structural_type, modes, characters |
| Referent Identifier(s) alternateIdentifier(s) | none besides the DOI | identifiers, relations (IsSameAs) |
| Registration Authority registrationAuthorityCode | Crossref — issued by Crossref (member 25296); held here as a copy | record.source_agency (our code, ra_doi_name, for names issued here once appointed) |
| Created Date issueDate | 2026-09-23 | record.registered (when the DOI name was first registered) |
| relatedIdentifiers | none needed — the descriptive metadata is in this record | container, relations (only where the descriptive metadata lives at another identifier) |
complete Every System Metadata element is here, with the basic metadata a journal article needs.
The System Metadata Declaration (JSON) · the Kernel Metadata Declaration (XML) · what each sub-type needs
Cited by — in this registry
No record held in this registry cites this DOI.
Counts only records held in this registry; a citation from a work held elsewhere is not seen here. The list as JSON
References with DOIs
The record lists 22 references; 22 carry a DOI.
| # | Reference | DOI | Held here |
|---|---|---|---|
| 1 | ref1 M. N. Armstrong and C. Trudeau, “Evaluating Architectural Extractors,” presented at Working Conference on Reverse Engineering, Honolulu, HI, pp. 30-39, October 12-14, 1998 | 10.1109/wcre.1998.723173 | not held here |
| 2 | ref2 Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach, “The Goal Question Metric Approach,” in Encyclopedia of Software Engineering, Two Volume Set, Gianluigi Caldiera and D… | 10.1002/0471028959.sof142 | not held here |
| 3 | ref3 Berndt Bellay and Harald Gall, “An Evaluation of Reverse Engineering Tool Capabilities,” Software Maintenance: Research and Practice, vol. 10, pp. 305-331, 1998 | 10.1002/(sici)1096-908x(199809/10)10:5<305::aid-smr175>3.0.co;2-7 | not held here |
| 4 | ref4 Ivan T. Bowman, Michael W. Godfrey, and Ric Holt, “Connecting Architecture Reconstruction Frameworks,” Journal of Information and Software Technology, vol. 42, no. 2, pp. 93-104, … | 10.1016/s0950-5849(99)00082-8 | not held here |
| 5 | ref5 Serge Demeyer, Tom Mens, and Michel Wermelinger, “Towards a Software Evolution Benchmark,” presented at Proceedings of the International Workshop on Principles of Software Evoluti… | 10.1145/602497.602502 | not held here |
| 6 | ref6 Martin S. Feather, Stephen Fickas, Anthony Finkelstein, and Axel van Lamsweerde, “Requirements and Specification Exemplars,” Automated Software Engineering, vol. 4, pp. 419-438, 1… | 10.1023/a:1008680612960 | not held here |
| 7 | ref7 Rudolf Ferenc, Susan Elliott Sim, Richard C. Holt, Rainer Koschke, and Tibor Gyimóthy, “Towards a Standard Schema for C/C++,” presented at Eighth Working Conference on Reverse Eng… | 10.1109/wcre.2001.957809 | not held here |
| 8 | ref8 Jim Gray, “The Benchmark Handbook: For Database and Transaction Processing Systems,” San Mateo, CA: Morgan Kaufman Publishers, Inc, 1991 | 10.1145/130951.1045493 | not held here |
| 9 | ref9 John L. Henning, “SPEC CPU2000: Measuring CPU Performance in the New Millennium,” IEEE Computer, pp. 28-35, July, 2000 | 10.1109/2.869367 | not held here |
| 10 | ref10 Richard C. Holt, Andreas Winter, and Andy Schürr, “GXL: Toward a Standard Exchange Format.,” presented at Seventh Working Conference on Reverse Engineering, Brisbane, Queensland, … | 10.1109/wcre.2000.891463 | not held here |
| 11 | ref11 Barbara Ann Kitchenham, “Evaluating software engineering methods and tools. Parts 1 to 12.,” ACM SIGSOFT Software Engineering Notes, vol. 21-23, 1996-1998 | 10.1145/290249.290255 | not held here |
| 12 | ref12 Thomas S. Kuhn, The Structure of Scientific Revolutions, Third Edition. Chicago: The University of Chicago Press, 1996 | 10.5897/ppr2013.0102 | not held here |
| 13 | ref13 William E. McUmber and Betty H. C. Cheng, “A Generic Framework for Formalizing UML,” presented at Proceeedings of the Twenty-third International Conference on Software Engineering… | 10.1109/icse.2001.919116 | not held here |
| 14 | ref14 Gail C. Murphy, David Notkin, William G. Griswold, and Erica S. Lan, “An Empirical Study of Static Call Graph Extractors,” ACM Transactions on Software Engineering and Methodology… | 10.1145/279310.279314 | not held here |
| 15 | ref15 Shari Lawrence Pfleeger, “Experimental Design and Analysis in Software Engineering, Parts 1-6,” ACM SIGSOFT Software Engineering Notes, vol. 19-20, 1994-1995 | 10.1145/224155.565630 | not held here |
| 16 | ref16 M. Plath and M. Ryan, “SFI: A Feature Integration Tool,” in Advances in Computer Science, R. Berghammer and Y. Lakhnech, Eds. Heidelberg, Germany: Springer Verlag, pp. 201-216, 19… | 10.1007/978-3-7091-6355-9_15 | not held here |
| 17 | ref17 Susan Elliott Sim, Richard C. Holt, and Steve Easterbrook, “On Using a Benchmark to Evaluate C++ Extractors,” presented at Tenth International Workshop on Program Comprehension, P… | 10.1109/wpc.2002.1021331 | not held here |
| 18 | ref18 Susan Elliott Sim and Margaret-Anne D. Storey, “A Structured Demonstration of Program Comprehension Tools,” presented at Seventh Working Conference on Reverse Engineering, Brisban… | 10.1109/wcre.2000.891465 | not held here |
| 19 | ref19 Susan Elliott Sim, Margaret-Anne D. Storey, and Andreas Winter, “A Structured Demonstration of Five Program Comprehension Tools: Lessons Learnt,” presented at Seventh Working Conf… | 10.1109/wcre.2000.891471 | not held here |
| 20 | ref20 Tirumale Sreemani and Joanne M. Atlee, “Feasibility of Model Checking Software Requirements: A Case Study,” presented at Proceedings of COMPAS'96, Gaithersburg, Maryland, pp. 77-8… | 10.1109/cmpass.1996.507877 | not held here |
| 21 | ref21 Walter F. Tichy, “Should Computer Scientists Experiment More?” IEEE Computer, pp. 32-40, May, 1998 | 10.1109/2.675631 | not held here |
| 22 | ref22 Ellen M. Voorhees and Donna Harman, “Overview of the Eighth Text REtrieval Conference (TREC-8),” presented at Text REtrieval Conference (TREC-8), Gaithersberg, Maryland, pp. 1-24,… | 10.6028/nist.sp.500-246.overview | not held here |
Relations
The record states no relation, and no record held here states one with this DOI.
Funding
The record names no funder.
Each funder's page lists every work held here that names it. ROR IDs open at ror.org; Funder Registry IDs resolve at doi.org.
History — the ledger
Every change to this DOI, in order, as it was recorded. Entries are only ever added, never changed or removed.
| # | When | What | By | Changes |
|---|---|---|---|---|
| 1 | 29 Sep 2026, 10:00 PM | register registered at Crossref; record read from api.crossref.org | Administrator (admin) | 124 fields set · sha256 8eea7b03488a… |
| 2 | 30 Sep 2026, 12:00 AM | update record re-read from api.crossref.org | Administrator (admin) | container.titles.0.value: |
Machine-readable: the history as JSON, with the full record after each change.
Changes made at Crossref through DOI Doctor
Every deposit Smart Scholars' DOI Doctor made for this DOI, oldest first, with Crossref's own answer. Each fingerprint is the SHA-256 of its entry and of the fingerprint before it, so a change to any earlier entry changes every fingerprint after it.
| When | What changed | Crossref | Fingerprint |
|---|---|---|---|
| 27 Sep 2026, 02:07 | Licence added https://creativecommons.org/licenses/by/4.0/ | ● accepted by Crossref | 94e078badc97… |
| 27 Sep 2026, 02:13 | Address updated from https://www.jst.org.in/index.php/pub/article/view/532/474 to https://www.jst.org.in/index.php/pub/article/view/532 | ● accepted by Crossref | a2bd500037a2… |
| 27 Sep 2026, 02:34 | Read back: the DOI resolves to the new address | ● confirmed | c2e53356b332… |
3 entries · the last fingerprint c2e53356b3328b276d187bd753e8721bbbacd425cead47d0790dcb7d7f37a70c · the history as JSON, each entry with the fingerprint before it, to check them yourself.



