AI in testing — Manual QA · QA Interview Hub

What is an LLM and how does it work in simple words?

An LLM (large language model) is a neural network trained on a huge amount of text to predict the next fragment of text (a token). It doesn't "know" facts the way a database does, but has learned the patterns of language: which words and ideas usually go together.

An analogy: a very advanced autocomplete. You give a prompt, and the model step by step adds the most likely continuation. All it "sees" at the moment is the context window: the prompt, the conversation history and attached files. So the quality of the answer depends directly on the context you gave.

In an interview: An LLM is a neural network that learned from a huge amount of text to predict the next word. It doesn't check facts, but generates the most likely continuation based on the context I gave it. Hence the main rule of working with it: the more precise the context, the better the result, and any answer must be verified.

How does AI help in QA's work?

Requirements analysis: find ambiguities, missed scenarios, questions for the PM.
Test design: drafts of test cases and checklists, boundary values, negative scenarios.
Test data: generate JSON, CSV, sets of invalid values.
Bugs: structure a bug report, analyse a stack trace or a log.
Automation: drafts of automated test code, locators, SQL queries; Copilot in the editor.
Documentation: a test plan, a report, release notes from a list of tasks.

What they all have in common: AI makes a draft faster, but the decision, verification and responsibility are QA's.

In an interview: I use AI at several stages: to find gaps in requirements, to get a draft of test cases with boundary and negative scenarios, to generate test data, to analyse a log and to write automated test code faster. But it is always a draft: I review it, complete it and answer for the result.

How do you write prompts to get decent test cases and not fluff?

Context: a description of the feature, acceptance criteria, constraints, business rules.
Role: "You are a Senior QA testing a payment form".
Techniques: "apply equivalence classes and boundary values, add negative scenarios".
Format: a table with the columns ID, preconditions, steps, expected result, priority.
An example of one good case, so the model repeats the style.
Iterations: "what did I miss on security?", "add network failure scenarios".

Bad:
  Write test cases for a transfer.

Good:
  You are a Senior QA on a fintech product. The feature: a card-to-card transfer.
  Rules: amount 1–10,000 UAH, 2 decimal places, daily limit 20,000,
  fee 1%, minimum 5 UAH.
  Apply equivalence classes and boundary values, add negative
  scenarios, double submission and a timeout.
  Format: a table ID | Preconditions | Steps | Expected result | Priority.
  Separately list the questions about requirements that remain unclear.

In an interview: I give the model context: a description of the feature, business rules and acceptance criteria, set a role, name the test design techniques and the result format, sometimes with an example. And at the end I ask it to list separately the unclear questions about the requirements, which is often the most valuable part of the answer. Then I iterate: I ask what is missing on security or failures.

What are hallucinations and why can't you take an AI answer as is?

A hallucination is when a model confidently produces something untrue: a nonexistent API method, an invented config option, a requirement that isn't in the document. The cause is that the model generates plausible text and doesn't check facts; when data is lacking, it fills the gap with the most likely variant.

What follows for QA:

Check generated cases against the real requirements: the model could have added its own "rules".
Run and review code as a colleague's code on a PR.
Check methods, commands and parameters against the documentation.
AI doesn't know your product, the history of bugs and the team's agreements, so coverage from it is never complete.

In an interview: A hallucination is when a model confidently produces something untrue, because it generates plausible text rather than checking facts. So I check everything generated against requirements and documentation, and I run and review code as a colleague's code. AI doesn't know our product and the history of bugs, so its cases are a starting point, not complete coverage.

What data must not be passed to AI tools?

Customers' personal data, real database dumps.
Passwords, tokens, API keys, configs with secrets.
The customer's code and documentation, unless the company has approved a specific tool.
Data covered by an NDA.

The rule: use only tools approved by the company (corporate plans where data doesn't go into training models), and anonymise everything you pass. If there is no policy, ask rather than assume.

In an interview: I don't pass personal data, secrets, or the customer's code and documentation into public AI tools unless it is approved. I work with tools the company has allowed, and I anonymise examples. And if there is no policy, I ask rather than decide myself.

How do you test features built on LLMs (a chatbot, an assistant)?

The main difference is non-determinism: for the same query the answer is a little different each time, so a check "the answer equals X" doesn't work.

Criteria instead of exact text: the answer contains the needed facts, doesn't contain forbidden ones, has the right language and tone, and the format meets requirements.
A set of reference queries (an eval set) with expected quality, run after every prompt or model change.
Prompt injection: attempts to make the bot ignore instructions, reveal the system prompt or other users' data.
Hallucinations: questions the bot must answer with "I don't know" or hand over to a human.
Boundaries: the bot gives no advice outside its role and takes no actions without confirmation.
Non-functional: response time, query cost, behaviour when the model is unavailable.

In an interview: LLM responses are non-deterministic, so I check not the exact text but criteria: whether the needed facts are there, whether forbidden ones are absent, whether the format and tone are right. I have a set of reference queries that I run after every prompt or model change. Separately I test prompt injection, hallucinations, and that the bot correctly says "I don't know" or hands the question to a human.

What are the limits of AI-generated test cases?

It doesn't know the product's context: integrations, incident history, team decisions.
It generates many obvious cases and few deep ones: races, states, unusual data.
It may invent rules that aren't in the requirements.
It duplicates cases and doesn't prioritise by risk.
Quality depends directly on the quality of the requirements: bad requirements give bad cases.

So a stronger approach is to ask AI not to "write cases" but to find gaps in the requirements and in the cases already written.

In an interview: AI doesn't know the product's context and the history of bugs, generates many obvious cases and few deep ones, may invent rules and doesn't prioritise by risk. So I more often use it as a reviewer: I ask it to find gaps in the requirements and in my cases, not to write everything for me.

Tell me how you use AI in your work.

This question is about your experience, so the answer must be concrete: tool → task → result → how you verified. The structure:

Which tools and where: Copilot in the editor, Claude or ChatGPT for analysing requirements and documentation.
A concrete task: for example, drafts of step definitions, test cases from acceptance criteria, PR review.
The result, preferably with a number: faster by X, less routine.
How you controlled quality: review, running, checking against requirements.
The limits: what you don't pass to AI and where it doesn't help.

Don't exaggerate: the interviewer will almost always ask for details, and the answer must come from your own experience.

In an interview: Right now I work in AI-driven automation: I use Copilot in the editor for drafts of automated test code and Claude for analysing requirements, generating test cases from acceptance criteria and reviews. This noticeably reduces routine, but everything generated I review, run and check against the requirements. I don't pass customer data and secrets to AI.

How is AI changing QA's role and will it replace testers?

AI takes over the routine: drafts of cases, data, code, a first analysis of logs and defect triage. QA's value shifts to what AI does poorly: understanding business context and risks, exploratory testing, critical thinking, communication with the team, and verifying AI systems themselves.

A balanced answer: AI won't replace QA, but a QA who uses AI effectively will replace one who doesn't.

In an interview: AI takes over the routine: drafts of cases, data and code, a first analysis of logs. And QA's value shifts to understanding risks and the business, exploratory testing, critical thinking and testing AI features themselves. I don't think AI will replace testers, but a tester who knows how to use it will be stronger than one who doesn't.

AI tools in QA's work

For code I use AI assistants in the IDE: GitHub Copilot, Cursor, Claude Code. They speed up writing automated tests, Page Objects, custom commands and refactoring. They also help to understand someone else's code and to find the cause of a test failure from a log.

Chat assistants (Claude, ChatGPT) I use for generating ideas for test cases and checklists from requirements, and for finding boundary and negative scenarios. Also for generating test data, SQL queries, JSON schemas and regular expressions. It is useful to ask AI to analyse requirements for contradictions and missed cases.

Other uses: writing bug reports and documentation, translation, self-healing locators in some platforms, analysing large logs. It is important to mention the limits too: I don't pass confidential data, tokens and users' personal data to external AI services.

Risks of AI-generated tests and code

AI can generate tests that check the wrong thing or that duplicate the current behaviour, bugs included. Invented APIs, methods or library options that don't exist, or outdated syntax, are common. Generated code may contain antipatterns: fixed waits, brittle selectors, no assertions.

More risks: superficial coverage of "happy" scenarios without edge cases, tests that always pass, and leaking confidential data to an external service. Also AI doesn't know the business context unless you provide it.

I validate like this: I read and understand every generated test, check it against requirements, and verify that it fails when the functionality is broken. The code goes through the same code review, linter and CI as hand-written code. The responsibility for the result stays with me, and AI is only a tool.

Testing a non-deterministic LLM feature

Since the answer is different every time, exact string comparison doesn't work, so we check the properties of the answer, not the specific text. For example: the answer contains the key information, answers the question, has the needed format, language and length, and doesn't contain forbidden topics. This is similar to a property-based approach.

A test dataset is needed: a set of typical, boundary and complex queries with the expected characteristics of answers. Each query is run several times and stability is assessed: for example, 9 out of 10 answers must be correct. The result is not "passed/failed" but a metric with a threshold.

Helper techniques: lowering temperature to 0 for reproducibility on a test environment, fixing the model and prompt version. I also check edge cases: an empty query, very long text, another language, an ambiguous question, an irrelevant query. And regression whenever the prompt or model changes is a must.

Hallucinations

A hallucination is a confident, plausible answer from a model that is factually wrong or not confirmed by sources. Examples: invented facts, quotes, links, function names, numbers, or an answer that contradicts the provided context. Especially critical in RAG systems, where the model must answer only from the documents.

I detect them manually by checking facts against ground truth: comparing with documentation, the knowledge base, system data. Provocative questions about what isn't in the context are useful: the right behaviour is to say "I don't know", not to invent. I also check that links and quotes really exist.

Automated: comparison with a reference answer from a golden dataset, a faithfulness check, whether every statement of the answer is supported by the context. For this LLM-as-a-judge and frameworks like RAGAS, DeepEval or promptfoo are used. Metrics: faithfulness, answer relevancy, context precision.

Quality criteria for an LLM answer

The main criteria: factual correctness, relevance to the question, completeness, conformance to context, following the format and instructions. Also tone and style matching the brand, language, length, safety (no harmful, biased or toxic content). For production, latency and query cost matter too.

A golden dataset is a reference set of queries with expected answers or criteria, prepared and verified by people. Every new version of a prompt or model is run on it and the metrics are compared. It works as a regression suite for an LLM.

A rubric is a clear evaluation scale, for example 1–5 for each criterion with a description of each level. LLM-as-a-judge is when another model evaluates an answer against a rubric, which makes evaluation scalable. But the judge must be calibrated: its scores are selectively checked against human ones.

Prompt injection and jailbreak

Prompt injection is an attack where an attacker inserts instructions that make the model ignore the system prompt. A direct one is in the user's message, for example "Ignore previous instructions and...". An indirect one comes through data the model reads: a web page, a document, an email, an API response.

A jailbreak is an attempt to bypass a model's safety restrictions to get forbidden content. Typical techniques: role play, hypothetical scenarios, text encoding, splitting a request into parts, another language. The goal of testing is to make sure the protections work.

How I test: a set of attack prompts from known catalogues, for example the OWASP Top 10 for LLM, checking for leaks of the system prompt and confidential data. I also check that the model doesn't perform dangerous actions through tools without permission. Automation is helped by promptfoo, Garak and other red-teaming tools.

Automating checks of LLM answers

The first level is deterministic checks: JSON structure against a schema, presence of required fields, length, language, keywords, absence of forbidden words or personal data. They are fast and cheap, so they run on every commit. This can also be done in an ordinary Cypress or Playwright test through the API.

The second level is semantic checks: comparing an answer with a reference through embeddings and cosine similarity with a threshold, for example 0.8. The third is LLM-as-a-judge with a rubric, which evaluates correctness, relevance and tone. For RAG, the metrics faithfulness and context relevance are added.

Because of non-determinism each case is run several times and the percentage of successful answers is counted. In tests you can lower the temperature, but the final evaluation must be on production settings. Results are tracked over time to see regression after a prompt or model change.