Why scraping GitHub yields fragmented data about crypto projects
The public API returns not development history, but a current snapshot of a repository: a list of branches, latest commits, and pull request status.
Authorship, dates, and the size of changes for older edits have to be pulled page by page, and any missed request leaves a gap in repository activity that cannot later be filled.
Rate limits turn this into a race. Without a token with an expanded quota and while watching secondary search limits, collection across a hundred projects stops halfway: some repositories get a full profile, while others get only the tail of recent weeks. Commit metrics for projects with different collection start dates are then incomparable.
The second problem is topology. A commit from a fork, mirror, or CI bot looks the same in the output as work by a core developer: rebase and squash rewrite authorship, force-push removes a branch from the common graph, and dependency updates generate hundreds of service commits.
Until such events are labeled separately, the activity graph spikes exactly when the team fixes the build or migrates the repository.
Incomplete history combined with branch merging yields numbers that cannot be used to compare projects with each other — yet these are exactly what estimates are most often built on.
We examine how this is solved: which collection scheme yields a reproducible activity profile and how to separate development from service noise.
What managed collection of repository activity provides
A well-established collection answers three questions: how much history we see, how fresh the numbers are, and what happens when the external source stops returning data. Each answer is measured, not felt.
Completeness of commit and release history
The collector goes through not only the main branch, but also tags, release branches, merges, authorship, and dates — the picture is built from all significant references in the repository on GitHub.
If you traverse only the tip, the metric will show a lull where work is actually happening in a maintenance branch. The output is a dataset where both one-off spikes and a steady maintenance rhythm are visible.
Data freshness and update lag
Updates run on a schedule with an overlap window: the last day is rebuilt, so late edits and rewritten history are not lost. The delay between an event and its appearance in the data mart is hours, not weeks: developer churn is visible before it is written about in reviews.
Headroom on request limits removes the main risk. When the quota is exhausted, collection does not stop but runs at a lower frequency; the queue catches up on missed data without gaps in the series.
Dataset quality no longer depends on one successful pass — metrics are comparable across projects and over time.
Approaches to GitHub scraping: traversal, webhooks, incremental sampling
Data extraction from GitHub is done in three modes, and they do not replace each other: full traversal gives history completeness, incremental sampling gives low-cost regular updates, and change notifications give freshness.
The choice depends on whether the full project history or only the delta since the last collection is needed.
| Data collection mode | When it fits | Data source limitations |
|---|---|---|
| Full traversal | Initial history load, reconciliation of gaps after a failure | Longest, hits request limits |
| Incremental sampling | Scheduled metric updates: delta by dates or cursor | Requires storing state and replaying the window |
| Change notifications | Reaction within minutes: commits, releases, issues, comments | Requires a public endpoint; event delivery is not guaranteed |
When which mode is advantageous
We run full traversal once when onboarding a project and repeat it selectively when a gap is visible in the metrics. Incremental sampling becomes the main mode: it fetches only changed areas and removes most of the load. Notifications complement it where latency, not completeness, matters.
Rate limit limitations and ways to work around them
The number of requests per hour is limited; burst request series and limits on fetching large pages trigger separately.
We wrap this as follows: reduce frequency, cache responses and request only what changed, spread the load across multiple tokens, pause and retry on failure — collection continues without hitting a wall.
Stages of implementing data collection about crypto projects
The order of work almost does not depend on the specific project: first source inventory, then the storage model, then reconciliation where activity is already known manually.
We arrange the implementation stages so that each step leaves a verifiable artifact — an export, a schema, a calculation. Then discrepancies are caught at the beginning, when rework costs hours, not weeks.
- Source audit: which events are available (commits, pull requests, discussions, releases, repository copies) and what the GitHub API returns given rate limits.
- Data schema: entities “project”, “repository”, “participant”, “event”, keys and duplicate merge rules, unified time, raw responses alongside calculated tables.
- Project directories: a canonical list with repository, token, and team linkage, handling renames, code migrations, and mirrors, manual edits with history.
- Loader: incremental collection by cursor, retries on failures, duplicate-free writes, run log with duration and volume.
- Metric calculation: activity by authors, share of unique participants, regularity and commit streaks, aggregation windows, separate rules for bots.
- Validation on reference projects: compare with manual counting for projects with known activity, investigate a discrepancy greater than a few percent down to the cause.
- Solution handover: regulations for updating directories, documentation on the data schema, instructions for adding projects, joint review with your team.
The data schema and directories remain with you: new metrics and sources are added without rebuilding the entire pipeline.
What we hand over: configurations, schemas, regulations
All artifacts are transferred to the customer's repository along with access rights — nothing remains on our side, and maintenance does not depend on us. By delivery, the collection pipeline works autonomously, and its state is visible without reading code.
Documentation and update regulations
- Collection configurations: list of sources, initial load depth, run schedule, project exclusion rules.
- Data schema: description of tables, keys, and relationships between commits, releases, participants, and version tags.
- Update regulations: polling frequency, behavior when GitHub response structure changes, procedure for onboarding a new project.
- Deployment documentation: how to bring up the pipeline from scratch and migrate it to other capacity without code changes.
- Analyst instructions: where activity metrics are stored and how to build custom queries.
Monitoring and alerts
- Checks for source availability and remaining request limits — the signal arrives before the export starts falling behind.
- Freshness control: if the latest record is older than the expected window, a notification triggers.
- Collection status panel: how many projects were processed per run, where failures occurred, how data volume changed.
- Error log with request context, so the cause of a failure can be found without contacting us.
Basic scenarios — restarting a run, adding a project, troubleshooting a failure — the customer handles independently. The maintenance team takes on the rest.
Case: how data completeness grew with the same request volume
The client is a crypto project analytics platform: investors look at team activity in repositories and make decisions based on it.
Data was collected by regular GitHub scraping, but the snapshot covered only 61% of required organizations — some projects looked “dead” despite live commits, and analysts corrected reports manually.
What it was before
Traversal ran over a flat repository directory: only the main branch, no forks or mirrors, without accounting for team moves to a new organization.
When rate limits triggered, the job failed entirely, and the next pass started only the next day — gaps of 2-3 days appeared in history, and period aggregates were calculated by last commit date instead of daily sums. The request failure rate stayed at 4.2%, and each failure had to be recovered manually.
What it became after
Collection was rebuilt to traverse by organizations with pagination and cursor persistence — an interrupted pass resumes from the same place, not from the beginning.
We added branch and fork parsing, an intermediate metadata store to avoid querying the source twice for the same thing, and request retries with increasing delay.
Coverage completeness grew from 61% to 94%, the failure rate dropped to 0.3%, and the number of source requests — that same request volume — increased by only 8%.
Full history rebuild decreased from 6 hours to 40 minutes, and the analytics summary no longer has projects with zero activity despite live commits.
How does collection via API differ from parsing project pages?
The GitHub API returns data in structured form: the field schema is documented, the response comes in machine-readable format, and request limits are known in advance.
For regular metric collection, this provides predictable load and low maintenance cost — the connector is written once and lives until the project itself changes the schema or access rules.
Page parsing works where there is no API or it does not return everything: contribution graph, participant list, monthly activity history.
The approach is more flexible, but relies on someone else's site markup — any change on the project side breaks parsing silently, returning zeros instead of an error.
Therefore such a source requires constant supervision: reference pages under tests, verification of control values, alerting when the event count drops.
The main selection criterion is not the speed of the first export, but source reliability and resilience to changes on the project side.
| Source | When it fits | Limitations |
|---|---|---|
| API | Standard metrics: commits, releases, participants, dependencies | Strict request limits, some fields closed |
| Webhooks | Need events at the moment of occurrence, without polling delay | Require a public endpoint and protection against duplicates |
| Page parsing | Aggregates and snapshots not available in the API | Breaks when markup changes, monitoring required |
We take the API as the basis, and page parsing as an auxiliary layer with convergence checks: if values diverge, an alert fires, and the metric does not go into the report.
Describe which project indicators are needed in the output — we will return with an estimate of source scope and a collection plan.
Technical doubts: limits, duplicates, historical data
We also recover project activity history retroactively: the public API returns paginated snapshots of commits, merges, and participants; daily snapshots and archives fill gaps where the output hits the depth limit.
Collection hits not access, but request frequency and data noise: without distribution over the window and caching of retries, the nightly run is lost entirely. Below are objections that come before kickoff.
Can you restore history for past periods?
Yes, but with varying accuracy. The deeper the period, the higher the chance that the output will not return data — we fill in with daily snapshots and archives, so on older sections the metric is reliable at the level of weeks, not hours.
How do you count contribution if a commit landed in several branches?
The key is the change hash plus parent: merges, repeated snapshots, and moves between branches collapse; a branch is not counted as a separate contribution. Otherwise activity is inflated many times over and the project ranking is distorted.
What to do with closed repositories?
They are publicly unavailable, so the report uses indirect signs — release tags and activity in related open mirrors. The share of closed work is marked as unknown, not as zero: otherwise a live project looks frozen.
Don't bots and mirrors distort the picture?
They do if not separated. We filter automatic commits and mirrors by author, regularity, and change type; we keep a separate list and show it broken down by repository.
Let's discuss your project data collection task
To return with an estimate and work plan, we need a minimum of inputs from you — usually we go through them in the first call.
The more specific the list of projects and questions for the data, the more accurate the estimate of effort and timelines: no need to budget for uncertainty.
- List of projects and repositories: organizations, mirrors, and forks that should be included or, conversely, excluded.
- Data sources: public GitHub via API, mirrors on other platforms, your internal tables and exports.
- Metric set: commit frequency, number of active contributors, merges, closed issues, releases, response speed to issues.
- History depth and period: from what date to collect, whether a one-time export or scheduled regular updates are needed.
- Output format: tables, database, transfer to your system, ready-made summaries and charts for reports.
- Quality requirements: how to filter out bots and inflated activity, what to do with renamed and archived repositories.
- Access and limitations: tokens with required permissions, source request limits, acceptable load on your services.
- Rules for storing and updating collected data, the person responsible for acceptance of the result on your side.
After that, we return with a timeline estimate, a stage-by-stage work plan, and a description of what you will get in the output. If some inputs are not ready yet — we will discuss which clarifications are critical and which can be resolved during the project.