Skip to content
Smart Scholars🛡 Scholar Shield🏛 Research Integrity Desk🧩 Portfolio Console📰 Journals🔧 DOI MembersTools🔎 Journal AuditGI GetIndexedDr DOI Doctor

10.46243/jst.2021.v6.i06.pp139-149 registered

Using Benchmarking to Advance Research: A Challenge to Software Engineering

Resolves to https://www.jst.org.in/index.php/pub/article/view/532

Held by Longman Publishers (India) · prefix 10.46243 live · DOI address https://doi.org/10.46243/jst.2021.v6.i06.pp139-149

Registered 29 Sep 2026 via crossref · record version 2 · last change 30 Sep 2026, 12:00 AM · record sha256 2cf2633011cd7cc0…

Resolve ⬇ Record (JSON) ⬇ Kernel Metadata Declaration (XML) Compare with Crossref Cite (APA · BibTeX · RIS · CSL)

Also as schema.org JSON-LD · Turtle · RDF/XML · DataCite XML · Crossref UNIXREF · Crossref UNIXSD — the formats of the DOI content-negotiation service, from this record (which carries what).

What the DOI identifies

JournalArticle — an article in a journal · Digital · Visual · en

Using Benchmarking to Advance Research: A Challenge to Software Engineering (PrincipalTitle · en)

Published 2021

Part of Journal of Science & Technology · ISSN 2456-5660 · volume 06 · issue 06 · pages 139–149

Agents

  • K Vijay Kumar (author)
  • Dr.Y V R Naga Pawan (author)
  • Longman Publishers (publisher)

Identifiers DOI 10.46243/jst.2021.v6.i06.pp139-149

Abstract · en

Benchmarks have been used in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. These observations have led us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. In support of this challenge, we present a case study of the reverse engineering community, where we have successfully used benchmarks to advance the state of research.

Licence https://creativecommons.org/licenses/by/4.0/

System metadata — ISO 26324:2025, Annex B · DOI Handbook 10.1

Each element by the standard's name (Annex B: reference elements, then administrative) and the Handbook's (in grey), read off the record above.

ElementValueIn the record
DOI Name
DOI name
10.46243/jst.2021.v6.i06.pp139-149doi
Referent Type
referentType
Creationreferent
Referent Sub-Type
referentSubType
JournalArticle — an article in a journaltype
Referent Name(s)
referentName(s)
Using Benchmarking to Advance Research: A Challenge to Software Engineering (PrincipalTitle, en)titles
Basic Metadata
basicMetadata
author: K Vijay Kumar
author: Dr.Y V R Naga Pawan
publisher: Longman Publishers
published: 2021
part of: Journal of Science & Technology · ISSN 2456-5660 · vol. 06 · no. 06 · pp. 139–149
language: en
form: Digital · Visual · Language
agents, dates, container, language, structural_type, modes, characters
Referent Identifier(s)
alternateIdentifier(s)
none besides the DOIidentifiers, relations (IsSameAs)
Registration Authority
registrationAuthorityCode
Crossref — issued by Crossref (member 25296); held here as a copyrecord.source_agency (our code, ra_doi_name, for names issued here once appointed)
Created Date
issueDate
2026-09-23record.registered (when the DOI name was first registered)
relatedIdentifiersnone needed — the descriptive metadata is in this recordcontainer, relations (only where the descriptive metadata lives at another identifier)

complete Every System Metadata element is here, with the basic metadata a journal article needs.

The System Metadata Declaration (JSON) · the Kernel Metadata Declaration (XML) · what each sub-type needs

Cited by — in this registry

No record held in this registry cites this DOI.

Counts only records held in this registry; a citation from a work held elsewhere is not seen here. The list as JSON

References with DOIs

The record lists 22 references; 22 carry a DOI.

#ReferenceDOIHeld here
1ref1 M. N. Armstrong and C. Trudeau, “Evaluating Architectural Extractors,” presented at Working Conference on Reverse Engineering, Honolulu, HI, pp. 30-39, October 12-14, 199810.1109/wcre.1998.723173not held here
2ref2 Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach, “The Goal Question Metric Approach,” in Encyclopedia of Software Engineering, Two Volume Set, Gianluigi Caldiera and D…10.1002/0471028959.sof142not held here
3ref3 Berndt Bellay and Harald Gall, “An Evaluation of Reverse Engineering Tool Capabilities,” Software Maintenance: Research and Practice, vol. 10, pp. 305-331, 199810.1002/(sici)1096-908x(199809/10)10:5<305::aid-smr175>3.0.co;2-7not held here
4ref4 Ivan T. Bowman, Michael W. Godfrey, and Ric Holt, “Connecting Architecture Reconstruction Frameworks,” Journal of Information and Software Technology, vol. 42, no. 2, pp. 93-104, …10.1016/s0950-5849(99)00082-8not held here
5ref5 Serge Demeyer, Tom Mens, and Michel Wermelinger, “Towards a Software Evolution Benchmark,” presented at Proceedings of the International Workshop on Principles of Software Evoluti…10.1145/602497.602502not held here
6ref6 Martin S. Feather, Stephen Fickas, Anthony Finkelstein, and Axel van Lamsweerde, “Requirements and Specification Exemplars,” Automated Software Engineering, vol. 4, pp. 419-438, 1…10.1023/a:1008680612960not held here
7ref7 Rudolf Ferenc, Susan Elliott Sim, Richard C. Holt, Rainer Koschke, and Tibor Gyimóthy, “Towards a Standard Schema for C/C++,” presented at Eighth Working Conference on Reverse Eng…10.1109/wcre.2001.957809not held here
8ref8 Jim Gray, “The Benchmark Handbook: For Database and Transaction Processing Systems,” San Mateo, CA: Morgan Kaufman Publishers, Inc, 199110.1145/130951.1045493not held here
9ref9 John L. Henning, “SPEC CPU2000: Measuring CPU Performance in the New Millennium,” IEEE Computer, pp. 28-35, July, 200010.1109/2.869367not held here
10ref10 Richard C. Holt, Andreas Winter, and Andy Schürr, “GXL: Toward a Standard Exchange Format.,” presented at Seventh Working Conference on Reverse Engineering, Brisbane, Queensland, …10.1109/wcre.2000.891463not held here
11ref11 Barbara Ann Kitchenham, “Evaluating software engineering methods and tools. Parts 1 to 12.,” ACM SIGSOFT Software Engineering Notes, vol. 21-23, 1996-199810.1145/290249.290255not held here
12ref12 Thomas S. Kuhn, The Structure of Scientific Revolutions, Third Edition. Chicago: The University of Chicago Press, 199610.5897/ppr2013.0102not held here
13ref13 William E. McUmber and Betty H. C. Cheng, “A Generic Framework for Formalizing UML,” presented at Proceeedings of the Twenty-third International Conference on Software Engineering…10.1109/icse.2001.919116not held here
14ref14 Gail C. Murphy, David Notkin, William G. Griswold, and Erica S. Lan, “An Empirical Study of Static Call Graph Extractors,” ACM Transactions on Software Engineering and Methodology…10.1145/279310.279314not held here
15ref15 Shari Lawrence Pfleeger, “Experimental Design and Analysis in Software Engineering, Parts 1-6,” ACM SIGSOFT Software Engineering Notes, vol. 19-20, 1994-199510.1145/224155.565630not held here
16ref16 M. Plath and M. Ryan, “SFI: A Feature Integration Tool,” in Advances in Computer Science, R. Berghammer and Y. Lakhnech, Eds. Heidelberg, Germany: Springer Verlag, pp. 201-216, 19…10.1007/978-3-7091-6355-9_15not held here
17ref17 Susan Elliott Sim, Richard C. Holt, and Steve Easterbrook, “On Using a Benchmark to Evaluate C++ Extractors,” presented at Tenth International Workshop on Program Comprehension, P…10.1109/wpc.2002.1021331not held here
18ref18 Susan Elliott Sim and Margaret-Anne D. Storey, “A Structured Demonstration of Program Comprehension Tools,” presented at Seventh Working Conference on Reverse Engineering, Brisban…10.1109/wcre.2000.891465not held here
19ref19 Susan Elliott Sim, Margaret-Anne D. Storey, and Andreas Winter, “A Structured Demonstration of Five Program Comprehension Tools: Lessons Learnt,” presented at Seventh Working Conf…10.1109/wcre.2000.891471not held here
20ref20 Tirumale Sreemani and Joanne M. Atlee, “Feasibility of Model Checking Software Requirements: A Case Study,” presented at Proceedings of COMPAS'96, Gaithersburg, Maryland, pp. 77-8…10.1109/cmpass.1996.507877not held here
21ref21 Walter F. Tichy, “Should Computer Scientists Experiment More?” IEEE Computer, pp. 32-40, May, 199810.1109/2.675631not held here
22ref22 Ellen M. Voorhees and Donna Harman, “Overview of the Eighth Text REtrieval Conference (TREC-8),” presented at Text REtrieval Conference (TREC-8), Gaithersberg, Maryland, pp. 1-24,…10.6028/nist.sp.500-246.overviewnot held here

The references as JSON

Relations

The record states no relation, and no record held here states one with this DOI.

The relations as JSON

Funding

The record names no funder.

Each funder's page lists every work held here that names it. ROR IDs open at ror.org; Funder Registry IDs resolve at doi.org.

History — the ledger

Every change to this DOI, in order, as it was recorded. Entries are only ever added, never changed or removed.

#WhenWhatByChanges
129 Sep 2026, 10:00 PMregister
registered at Crossref; record read from api.crossref.org
Administrator (admin) 124 fields set · sha256 8eea7b03488a…
230 Sep 2026, 12:00 AMupdate
record re-read from api.crossref.org
Administrator (admin)
container.titles.0.value: Journal of Science &amp; Technology → Journal of Science & Technology

Machine-readable: the history as JSON, with the full record after each change.

Changes made at Crossref through DOI Doctor

Every deposit Smart Scholars' DOI Doctor made for this DOI, oldest first, with Crossref's own answer. Each fingerprint is the SHA-256 of its entry and of the fingerprint before it, so a change to any earlier entry changes every fingerprint after it.

WhenWhat changedCrossrefFingerprint
27 Sep 2026, 02:07Licence added
https://creativecommons.org/licenses/by/4.0/
● accepted by Crossref94e078badc97…
27 Sep 2026, 02:13Address updated
from https://www.jst.org.in/index.php/pub/article/view/532/474 to https://www.jst.org.in/index.php/pub/article/view/532
● accepted by Crossrefa2bd500037a2…
27 Sep 2026, 02:34Read back: the DOI resolves to the new address● confirmedc2e53356b332…

3 entries · the last fingerprint c2e53356b3328b276d187bd753e8721bbbacd425cead47d0790dcb7d7f37a70c · the history as JSON, each entry with the fingerprint before it, to check them yourself.

Share this page

Share this page

Ask AI about this page

Everything Smart Scholars runs

Nine sites, one account. A journal starts at the audit; an author starts at Scholar Shield.

What we do: DOIs registered from what a journal already has · Crossref records completed — references, abstracts, licences, ORCID iDs, affiliations and ROR IDs, funding · DOIs that resolve · articles preserved · journals rated on the public record · see everything we do →

Book a free call

For journals & publishers

Start with the audit — ₹4,999 per journal, and the gate to everything else.

DOI care

Every service on one journal profile — each previews first and acts only on your approval.

  • DOI Doctor — check & repairevery DOI tested at doi.org and Crossref; dead links, abstracts, licences, ORCIDs
  • Smart Deposita new DOI from your PDF, Word file, article page, JATS XML or spreadsheet — every value shown with its source, deposited once you confirm it
  • DOI census (Excel)site against Crossref, both directions, with a copy-paste fix per article
  • Backfile DOIsDOIs for older articles that never had one — no per-DOI fee of ours
  • DOIs for conference proceedingsthe conference, the volume and every paper — registered in the organiser's own name
  • Scopus indexing guidethe minimum criteria, the 14 CSAB criteria and how a title is suggested — from Elsevier's own page
  • Embase indexing guidewhat Embase asks of a biomedical journal and how to suggest it
  • Write DOIs into OJSthe verified DOIs pushed onto your article pages through your own API
  • Silent-failure monitorweekly checks of site, OAI-PMH, Scholar tags, ISSN, preservation, hijack lists
  • Journal Radarevery journal, every day: SSL, DNS, redirects, robots.txt, OJS, DOIs, Crossref metadata, broken links — what broke since yesterday first
  • Verified metrics & badgesa public, dated, source-linked metrics page and a "cited by" badge
  • Quality dossier (PDF)one PDF for universities, authors and reviewers
  • Author citation enginethe citation steps emailed to each author from the journal's own mailbox
  • JATS XMLmachine-readable JATS per article
  • Duplicate articlesthe same paper published twice — you choose which copy stays

For authors & researchers

Free to use. Nothing you check is shared with the journal.

For institutions, sponsors & DOI operators

Smart Scholars

Mon–Sat, 10:00–19:00 IST. The Ask AI button on every page answers about our services at any hour.

News

Policies

What we can register a DOI for

20 kinds of record, one account, one place. Every one gets a DOI that resolves, metadata that indexes read, and a record that stays correct afterwards.

Journals

  • Journal articles
  • Journal titles
  • Pending publications
  • Peer reviews
  • Preprints & posted content

Books & conferences

  • Books
  • Book chapters
  • Book series
  • Book sets
  • Conference proceedings
  • Proceedings series
  • Conference papers

Other research output

  • Theses & dissertations
  • Reports & working papers
  • Report series
  • Standards
  • Databases
  • Datasets
  • Figures, tables & supplements

Funding

  • Grants & funding awards

Elsewhere

The same company, in the places our publishers already read.
Add Smart Scholars as a preferred source on GoogleSee more of our newsroom in Google’s Top Stories.
Smart Scholars · Every service on one pageNewsroom posts CC BY 4.0 · DOI metadata CC0 · everything else ©Data from OpenAlex (openalex.org), CC0 · Crossref · ISSN Portal · DOAJContact
WhatsApp