Why plant recognition in the app fails
Plant recognition breaks not in the lab but in the field: a user photographs a leaf on the move, autofocus cannot keep up with the wind, the frame reaches the model blurred — and the app confidently names the wrong species.
A single wrong result on a walk costs more than the absence of the feature itself: trust in the app is not restored after that.
Poor lighting is the second most common cause of failures. At dusk, against the light, or on the variegated background of undergrowth, leaf color and texture change more than the differences between closely related species, and species identification accuracy drops.
If preprocessing does not equalize exposure and separate the object from the background, the model compares noise with features.
A separate story is app operation without a network connection. There is no connection in the forest, so recognition must run on-device, but the entire species catalog does not fit there.
Model compression speeds up the response and at the same time cuts rare species — the very ones for which the app is installed.
A long response after shooting usually means computations run in the same thread as the camera: the interface freezes, especially on a weak device. An overgrown catalog makes the picture worse — the more species, the higher the chance of confusing similar ones. Below we break down how each of these failures is addressed.
What changes after implementation: response time and accuracy
The first measurable effect is response time to a photo. The model computes directly on-device, so the result appears within fractions of a second after the frame, without waiting for a server. On a weak connection, the difference reads as "instantly" instead of "wait."
The second is the share of correct identifications.
It is visible on a holdout sample broken down by season and region: baseline quality plus fine-tuning on user images raises accuracy on popular species, while rare and disputed cases go to server-side verification with response refinement.
The third is identification without a network.
The class of failures "no connection, try later" simply disappears: the reference guide and model reside inside the app, so recognition works in the forest, in the park, and in an underpass, and history syncs later when the connection returns.
The fourth is the load on app support. Some requests phrased as "it did not recognize" are closed by a hint directly in the app: typical questions leave the first line, leaving real errors and requests to expand the reference guide.
The fifth is stability on weak devices. The model is selected for the memory capacity of budget devices, so recognition does not cause crashes and restarts, and the response remains predictable across the entire device fleet.
We record these points as acceptance criteria: for each there is a measurement before and after implementation, so the effect is verified with numbers, not feelings.
Plant recognition schemes: on-device and in the cloud
The identification scheme determines not only accuracy but also what the team can change without releasing a new app version.
The limitations of the options are of different natures: in some cases it is connection and response latency, in others it is build weight, and in yet others it is someone else's rules for handling your images.
There are three forks: where the model computes, whose species catalog is used, and who owns the uploaded photos. Below are four working options and their limitations.
| Option | When it fits | Limitations |
|---|---|---|
| On-device model | Shooting without a network, response needed at the moment of the frame, stable species catalog | App size grows, inference is slower on weak devices, a new model only with a release |
| Server-side model inference | Broad species catalog and frequent model updates without rebuilding the app | Connection required, response depends on the channel, load grows with the number of users |
| Hybrid identification scheme | Common species computed on-device, rare ones refined on the server | Two logic branches and two sets of checks, harder to reproduce user complaints |
| Fine-tuning on your own images | Own or local species catalog, shooting in non-standard conditions | A labeled image set is required, regular retraining cycle and quality control |
| Ready-made external solution | Fast launch and coverage of popular species needed | Catalog and accuracy are set by the provider, your images and fine-tuning depend on the service terms |
The options do not exclude each other: server-side inference can conveniently remain a refinement point for rare species, while fast selection is kept on-device.
We lock in the choice after reviewing your catalog and shooting conditions — both accuracy and scope of work depend on them.
Implementation stages: from image audit to model handover
The order of work is structured so that recognition quality is confirmed before release, not after user complaints. Each stage closes with a verifiable artifact — a report, scheme, or labeled set — so the argument "works or not" is resolved by facts.
-
Image audit. We look at what actually enters the frame: angles, lighting, blurred and cropped frames, the share of leaves without flowers and fruits. We calculate the distribution by species — it is almost always uneven, and the training strategy depends on this.
-
Quality metric definition. We fix what we consider a correct answer: the exact species in the first suggestion or in a list of three, how we handle uncertain answers. The metric is agreed before training so the result is not later adjusted to convenient numbers.
-
Solution scheme. We determine where the model computes — on-device for offline operation or on the server, how the cache is arranged, and what we show on a weak connection. Here we also decide how to add new species without re-releasing the app.
-
Data labeling. Disputed species are labeled by botanists, and discrepancies are resolved with a second pass. We separately mark frames where the species is fundamentally undeterminable — otherwise the model learns to guess by the background.
-
Training and holdout image set. We assemble the holdout set by regions, seasons, and camera types so that it does not overlap with the training set. It reveals errors by class and the share of refusals, when it is better not to give an answer at all.
-
Model and documentation handover. We hand over the code, weights, retraining script, and instructions for expanding the species reference guide. The customer's team then updates the model and adds species independently.
Handover package: trained model, labeling, regulations
Handing over the app is not the moment when the project is ready to live without us. We transfer everything needed for plant recognition to develop through the customer's team: model weights, an image corpus with labels, app operation schemes, and regulations. Each item below is a working artifact, not a description of "roughly how it is arranged."
- Trained model for plant recognition in a portable form: weights, input configuration, and a list of supported species. Fine-tuning or replacement does not require rebuilding the app.
- Image labeling: the source corpus by species, divided into training and validation samples, plus labeling rules — which angles and lighting we consider valid, what we mark as an "uncertain case."
- App operation scheme: the path of a photo from camera to model response, preprocessing points, and a unified response format, including the refusal "could not recognize."
- Model update regulation: frequency of fine-tuning on new images, minimum volume of added data to restart training, and acceptance criteria for the share of correct answers on the validation sample.
- Version rollback rules: where previous weights are stored, how to switch to the previous version, and at which quality metrics the switch occurs without manual intervention.
- Support instructions: typical requests, log reading, procedure for a drop in recognition quality, and an escalation matrix.
- Accesses and rights: repository, image storage, and accounts are transferred with roles; whether we retain access after handover is decided by the customer.
We record the handover scope in an act so that nothing comes to light after the work is closed.
Case: identifying species and diseases from a field image
How it was before implementation
The user sent an image to support, the operator checked it against the reference guide manually and returned with a text answer.
Average response time was 6-8 hours; during the seasonal peak it reached up to a day, and clarifying questions ("photograph the leaf again," "a frame from the other side is needed") sent the person into a second waiting round. The share of such follow-up questions stayed at about one third of requests.
What was changed
Plant recognition was moved into the app itself: the image is processed on-device, and the reference guide remains the source of descriptions and care recommendations. The foundation is the open community model PlantNet, fine-tuned on user images and local flora.
The confidence threshold was configured so that with a weak result the app itself suggests rephotographing the leaf or inflorescence — before contacting support, not after.
What changed in numbers:
- Time to answer: from 6-8 hours to 2-4 seconds right on the screen.
- Share of clarifying follow-up questions: from 31% to 9% — mostly rare crops remained.
- The load on the first support line fell fourfold, and the freed-up engineers began handling complex cases.
- Accuracy on a test set of 2,000 images — 94% for species and 88% for disease signs.
User response: the share of those who return to shooting again grew by 40%, app store ratings — 4.6 versus 4.1. Such task analysis removes the main part of the support load even before the person writes a request.
What to do if a third-party model is already in the app
A third-party model in the app is not a death sentence, and there is no need to rewrite the app because of it. First we break down where exactly quality is lost: the model itself, frame preprocessing, the set confidence threshold, or the absence of classes for local flora.
Often the issue is not the model but that it was trained on foreign flora and simply does not see the species that grow where your users shoot.
Fine-tuning instead of training from scratch gives results noticeably faster and on a smaller data volume. We take the already working model as a foundation, preserve general features — leaf shape, venation, texture, growth habit — and add our classes on top.
A class needs several hundred confirmed images instead of tens of thousands, and plant recognition quality on typical species does not drop.
You most likely already have labeling, and it needs to be transferred, not collected anew. User-confirmed identifications, edit history, images rejected by moderation, geotags, and seasonality — all of this is converted into a training set.
Such material is more valuable than synthetic: it was shot on the same cameras and under the same conditions as real requests, so the model does not diverge from what the app sees.
We maintain compatibility with the current app at the contract level: if the input is a frame and the output is a list of species with confidence, the interface, analytics, and offline mode remain as they are. The model changes behind a wrapper layer, not inside the screens.
If the response format of the new model is different, we write field mapping — this is cheaper, faster, and safer than reworking the camera and user scenarios.
Fully replacing a third-party model is justified when the provider closes access, limits request volume, or quality hits the ceiling of someone else's model. Then we move computations to our side and preserve the same responses for the app.
Describe the task — we will return with an estimate and plan: what we fine-tune, what we migrate, and what stays in place.
Technical concerns: shooting, labeling, data volume
Field shooting is rarely ideal: backlight, a blurred frame, a single leaf without an inflorescence. Therefore the model is trained on a mixed set — open herbarium and amateur images plus frames shot under the same conditions as the user's.
Photo identification accuracy depends not on optics but on how many species are in the catalog and how close their features are: Pl@ntNet publishes metrics openly for tens of thousands of species, and they show where confusion between similar species begins.
An image is also a geotag, so by default only the frame itself is sent to the server, without metadata, while coordinates and identification history remain on-device.
We do not create labeling from scratch: some classes are covered by open identifications, and disputed frames go to a queue for review by a biologist. The species catalog is expanded without rebuilding the app — a new class is connected on the service side.
Does a user shooting at dusk make photo recognition worse?
The frame is cropped to the detected plant, and lighting and angle variants are added to the training set.
If confidence is below the threshold, the app does not substitute the wrong species but asks to photograph the leaf or flower again — the share of false answers drops, and the scenario does not break.
Where do user photos go?
A frame without metadata and geotags is sent for recognition; the walk route and identification history remain on-device. The image enters the training set only with explicit consent; otherwise it is deleted after the response.
How long will it take to train a model for our flora?
The base model on open data launches immediately and already provides working identification.
Fine-tuning for local species proceeds in iterations — the version updates every two to four weeks, and the catalog is expanded through the server-side reference guide, without an app update.
Let's discuss the task: what to clarify before work starts
Estimates and the work plan rest on facts, not guesses. Below is what we clarify before calculating timelines: each item noticeably changes the scope of work if left until the acceptance stage.
- Shooting scenarios: a single leaf close-up, a flower in the crown, seedlings near the ground, a single frame or shooting on the move — the model's dataset depends on this.
- Species catalog: region, number of species, whether cultivated varieties and rare plants are needed, whether we add cards with descriptions and care.
- Platforms and device fleet: minimum OS version, old models with a slow camera — affects the choice of recognition method.
- Offline operation: we compute the frame on-device or send it to the server, whether the catalog updates without releasing a new app version.
- Privacy: whether photos and geotags remain on the phone, what goes to the server, how and at whose request it is deleted.
- Response format: one name, a list of probable options with confidence, or a species card — determines the interface and hint logic.
- Start procedure: who provides test frames and labels, by which indicators we accept the result.
Let's discuss the task within these boundaries — we will return with an estimate and a stage-by-stage work plan so that you understand where each check stands and what you will get at the output.