Article Text


Clinical performance evaluation of the Idylla NRAS-BRAF mutation test on retrospectively collected formalin-fixed paraffin-embedded colorectal cancer tissue
  1. Louise Johnston1,
  2. Michael Power1,
  3. Philip Sloan2,3,
  4. Anna Long2,
  5. Angela Silmon4,
  6. Ben Chaffey4,
  7. Andrea Jane Lisgo4,
  8. Liam Little4,
  9. Ellen Vercauteren5,
  10. Torben Steiniche6,
  11. Tine Meyer6,
  12. John Simpson1
  1. 1National Institute for Health Research (NIHR) Diagnostic Evidence Co-operative (DEC) Newcastle, Newcastle University, Newcastle upon Tyne, UK
  2. 2Department of Cellular Pathology, Newcastle upon Tyne Hospitals NHS Trust, Newcastle upon Tyne, UK
  3. 3Northern Institute for Cancer Research, Newcastle University, Newcastle, UK
  4. 4NewGene Ltd, Bioscience Building, International Centre for Life, Newcastle upon Tyne, UK
  5. 5Biocartis NV, Generaal De Wittelaan, Mechelen, Belgium
  6. 6Institute of Pathology, Aarhus University Hospital, Aarhus, Denmark
  1. Correspondence to Dr Michael Power; Michael.Power{at}


Aims Understanding the molecular mechanisms of underlying disease has led to a movement away from the one-drug-fits-all paradigm towards treatment tailored to the genetic profile of the patient. The Biocartis Idylla platform is a novel fully automated, real-time PCR–based in vitro diagnostic system. The Idylla NRAS-BRAF mutation test has been developed for the qualitative detection of mutations in NRAS and BRAF oncogenes, facilitating genetic profiling of patients with cancer. The aim of this study was to carry out a formal clinical performance evaluation.

Methods Two-hundred and forty-two formalin-fixed paraffin-embedded (FFPE) human malignant colorectal cancer (CRC) tissue samples were identified in departmental archives and tested with both the Idylla NRAS-BRAF mutation test and the Agena Bioscience MassARRAY test.

Results The overall concordance between the Idylla NRAS-BRAF mutation test and the MassARRAY comparator reference test result was 241/242 (99.59%, lower bound of one-sided 95% CI=98.1%) for NRAS and 242/242 (lower bound of 95% one-sided 95% CI=98.89%) for BRAF. The Idylla NRAS-BRAF test detected one NRAS mutation that had not been reported by the MassARRAY comparator reference test. Reanalysis of this sample by droplet digital PCR confirmed that the mutation was present, but at an allelic frequency below the stated sensitivity level of the MassARRAY system.

Conclusion These results confirm that the Idylla NRAS-BRAF mutation test has high concordance with a widely used NRAS-BRAF test, and is therefore suitable for use as an in vitro diagnostic device for this application.

  • colorectal cancer
  • molecular oncology
  • molecular pathology

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from


The epidermal growth factor receptor (EGFR) and its downstream signalling pathways are involved in the development and progression of several human tumours including colorectal cancer (CRC).1–3 Activated signalling pathways including the RAS-RAF-BRAF-MAPK and the phosphatidylinositol 3-kinase (PI3K)-Akt pathways are key drivers of tumour growth and progression.4 5 Patients with CRC can be successfully treated with anti-EGFR monoclonal antibody therapies, such as panitumumab or cetuximab, which work by disrupting EGFR signals6–10—provided the tumours do not have mutations in EGFR signalling pathways.11–14 Testing patients for specific mutations that are linked to lack of response to EGFR blockade prior to commencing therapy ensures they receive the most appropriate treatment, and that treatment benefits, adverse effects and costs will be optimised.

Reduced anti-EGFR treatment efficacy was initially thought to be the result of mutations within KRAS exon 2 (codons 12 and 13). Therefore, KRAS genotyping was routinely used to stratify patients with CRC into those likely to respond to anti-EGFR therapy and those unlikely to respond. Further mutations causing resistance to anti-EGFR therapy were subsequently discovered through two large clinical studies (PRIME and CRYSTAL) which showed that 14%–17% wild-type (WT) KRAS exon 2 individuals actually responded poorly to anti-EGFR therapy.9 15 It is now known that resistance to therapy has a range of genetic causes: mutations in KRAS (exons 2, 3 and 4) and NRAS (exons 2, 3 and 4) are found in 40%–45% and 5%–10% of patients, respectively.5 16 17 BRAF V600E mutation is detected in 4%–18% of CRC cases and is prognostic of outcomes, but, in contrast to KRAS which is predictive of response is not used to guide anti-EGFR treatment.16–19

The Idylla System (Biocartis, Mechelen, Belgium) is a fully automated, real-time PCR-based molecular diagnostics system.20–26 The Idylla NRAS-BRAF mutation test is a novel in vitro diagnostic medical device (IVD) for the qualitative detection of mutations in codons 12, 13, 59, 61, 117 and 146 of the NRAS oncogene and codon 600 in the BRAF oncogene in formalin-fixed paraffin-embedded (FFPE) human malignant CRC tissue. This non-randomised study, was a clinical performance evaluation of the Idylla NRAS-BRAF mutation test with retrospectively collected samples and prospectively planned data generation.

Materials and methods

The Idylla platform

The Idylla molecular diagnostic platform is a fully automated, allele-specific real-time PCR-based molecular diagnostic system.20 It combines sample preparation with PCR thermocycling to detect molecular targets from a variety of solid and liquid samples including FFPE slices. Briefly, FFPE tissue sections are placed directly into the Idylla system cartridge, with no preprocessing. The cartridge is then inserted into the Idylla platform, where high-intensity focused ultrasound (HiFu) technology, buffers, reagents and heat induces deparaffinisation, tissue disruption, cell lysis and subsequent release of nucleic acid. Allele-specific primers and fluorescent probes then perform real-time PCR amplification and mutation detection. All required consumables are provided in the cartridge, and the Idylla Console and instruments are CE marked.

The Idylla NRAS-BRAF mutation test

The Idylla NRAS-BRAF mutation test uses allele-specific multiplex PCR reactions to amplify 23 mutations in codons 12, 13, 59, 61, 117, 146 of the NRAS oncogene and codon 600 of the BRAF oncogene. A set of parameters describing the generated PCR curves are determined by the Idylla software, for example, ΔCq value (calculated as the difference between the quantification cycle value (Cq) of the gene control signal and the Cq of the mutant signal). A sample is classified as mutation positive if the parameters of the PCR curve generated are within the validated range. Otherwise the sample is reported as being mutation negative, that is, WT. Table 1 lists the mutations detected by the test.

Table 1

NRAS and BRAF mutations detected by the Idylla NRAS-BRAF mutation test

The results obtained from the Idylla NRAS-BRAF mutation test during this study were not used for clinical or diagnostic purposes and were not made available to patients or their clinicians.

The comparator test

The NRAS and BRAF mutation status for each FFPE tissue sample was assessed using the MassARRAY System (Agena Bioscience). Existing test results were used if the comparator test had been performed within 2 years of the start of this study and no further slides or slices had been taken from the source FFPE block. H&E slides were taken at the start and end of block slicing to confirm sample homogeneity. If the sample block had been cut since the initial comparator test analysis, newly prepared alternating slides/slices were taken to rerun the comparator test. To reduce the risk of bias, different operators performed testing on the MassARRAY System and the Idylla NRAS-BRAF mutation test.

The Agena MassARRAY mutation test

The Agena MassARRAY mutation test used in this study is a Laboratory Developed Test which is within the scope of the ISO 15189 accreditation held by NewGene Ltd, and is routinely used clinically. Samples with known mutation profiles, as used during clinical validation of the assay for ISO 15189 accreditation, were used as positive controls. ‘No template’ reactions, prepared with deionised H2O in place of template DNA were used as controls for non-specific amplification.

MassARRAY utilises the primer extension analytical method to determine the presence or absence of single nucleotide polymorphisms (SNPs) which cause specific mutations in the protein encoded by a gene. First, target region-specific PCR with short amplicons is used to amplify the DNA encoding the target sequence. These PCR products are then probed for the presence of the mutation of interest using an oligonucleotide which is complementary to the region immediately adjacent to the SNP location. This is then extended by only one base, using 2',3' dideoxynucleotides (ddNTPs); the ddNTP which terminates the oligonucleotide chain corresponds to the base code at the SNP location. A matrix-assisted laser desorption/ionization time-of-flight (MALDI-ToF) mass spectrometry system is then used to differentiate between oligonucleotide species on the basis of their differing masses, identifying the various SNP bases present at that location and also quantifying the relative allelic frequency of the mutant sequence versus the WT.

Study size estimation

The number of samples required for a diagnostic agreement study with 80% power and expected agreement rate of 95% is 188. To allow for invalid runs and samples, the target number of FFPE tumour tissue blocks to be tested was set at 250. At least 30 valid FFPE samples with known NRAS mutations, at least 30 samples with known BRAF mutations, and up to 190 samples with WT KRAS were planned to be included.

Tissue selection criteria and process

The inclusion criteria for samples can be found in box 1. Samples were obtained from two centres: The Royal Victoria Infirmary, Newcastle upon Tyne Hospitals NHS Foundation Trust (UK; n=113), and the Institute of Pathology, Aarhus University Hospital (Denmark; n=156). All available samples were entered into the study. Figure 1 shows how FFPE tissue samples were prepared for use in the Idylla NRAS-BRAF mutation test.

Figure 1

Formalin-fixed paraffin-embedded (FFPE) sample preparation algorithm prior to Idylla NRAS-BRAF mutation test. Histologically confirmed, retrospectively collected FFPE colorectal cancer tissue samples were identified and tissue sections 5 or 10 µm thick were sampled as close as possible (within the same FFPE block) to the sections used for reference testing. Prior to analysis, the tumour content and area were determined on a H&E slide by a consultant histopathologist at both sites. If required, macrodissections were performed to ensure tumour nuclei content of more than 10% in the sample for analysis. The volume of material was specified to ensure that there was sufficient for the assay while not overloading the device.

Box 1

Sample selection criteria

Formalin-fixed paraffin-embedded (FFPE) colorectal cancer (CRC) samples were included if they were:

  • Obtained from male or female patients≥18 years of age

  • Approved for investigational purposes according to the applicable laws

  • Histologically confirmed to be primary or metastatic malignant CRC

  • Fixed in formalin for a maximum time of 48 hours

  • Less than 10 years old and had been stored at ambient conditions

  • Stored at room temperature for not more than 60 days prior to Idylla testing

CRC samples were excluded if:

  • The patient declined to provide their sample to third parties for research purposes

The protocol specified that samples should have at least 10% of their content identified as tumorous cells because the clinical importance of results (concordant or discrepant) from samples with very low proportions of tumorous cells could be debatable.

Diagnostic performance measurements

Baseline testing

No discrepancies were observed between the original results and the repeat results when retesting with the MassARRAY was performed on specimens that had originally been tested with other devices.

Percentage agreement

As it is not feasible to set-up a study with large numbers of each individual mutant, the performance of the Idylla NRAS-BRAF mutation test was scored based on total agreement. Overall agreement (% total agreement), and negative and positive agreements were determined together with their 95% one-sided CIs. Overall agreement was defined as the proportion of concordant results in all results. Positive agreement was defined as the proportion of concordant results in positive Idylla results. Negative agreement was defined as the proportion of concordant results in negative Idylla results.

Discordant by design results

The Idylla NRAS-BRAF panel detects 18 NRAS mutations and 5 BRAF mutations (table 1), while the MassARRAY NRAS-BRAF panel detects 25 NRAS mutations and provides a BRAF V600E result as a generic call rather than distinguishing between V600E/D or V600K/R. The seven additional NRAS mutations detected by the MassARRAY but not by the Idylla NRAS-BRAF mutation test panel include G12R, G13C, G13S, G13A, Q61P and Q61E. Samples that were anticipated to produce discordant results because of design differences in the Idylla and MassARRAY systems were classed as ‘discordant by design’ and were excluded from analysis.

Investigation of discordant results

Discordant results were further investigated by using droplet digital PCR as a second comparator test to evaluate the accuracy of the Idylla result. Droplet digital PCR was performed on a QX100TM system (Bio-Rad Laboratories, Hercules, California, USA) according to the instructions of the manufacturer.


Samples included

Two hundred and sixty-nine FFPE CRC samples were initially identified for the study (figure 2), and 242 samples included in the performance evaluation. Twenty-seven FFPE samples were excluded because they failed to meet the inclusion criteria. The most common reason for exclusion was insufficient sample quantity (n=11). One FFPE sample was excluded because it was classified as discordant by design. Seven samples resulted in a failed run, three of which were due to insufficient sample (assay failure) and four of which were due to instrument errors (system failure). This equates to assay and system failure rates of 1.24% and 1.74%, respectively. All invalid and error runs were successfully repeated and included in the agreement analysis as specified in the protocol. Samples included in the study ranged from blocks of tumour from resections (198) to small diagnostic core biopsies (44). Macrodissection was required in 126 samples. The proportions of tumour ranged from 20% to 60% in sections without macrodissection, and from 10% to 75% in sections requiring macrodissection.

Figure 2

Process of enrolling samples into the study.

The photomicrographs in figure 3 are from samples which show extensive mucinosis (A) extensive necrosis (B) and a small core biopsy (C). The Idylla NRAS-BRAF mutation test was able to detect mutations in all such samples.

Figure 3

Photomicrograph of tumour samples used for performance evaluation study. (A) sample with extensive mucinosis; (B) sample with extensive necrosis; (C) sample from small core biopsy of metastatic colon adenocarcinoma in the liver. The core was divided lengthways before embedding and the total section area was 25 mm2, the minimum specified in the protocol. The sample tested positive in both platforms without macrodissection and included solid adenocarcinoma (top left) normal liver (top right) and mucinous stroma (lower core).

Baseline demographic and clinical characteristics

The final study population demographics and clinical characteristics are presented in table 2. Gender representation was balanced between males and females (54% and 46%, respectively). The mean age at tumour collection was 67.6 years (range: 24–96 years). The majority of samples (91%) were of primary tumour tissues originating from the colon. The remaining samples were from metastatic deposits originated in the liver (3%) or another (6%) location. All samples were histologically confirmed as malignant CRC; 85% were KRAS WT, 15% had an NRAS mutation and 21% had a BRAF mutation.

Table 2

Baseline demographic and clinical characteristics for participants with valid test results

Clinical performance of the Idylla NRAS-BRAF mutation test

The NRAS and BRAF mutational status of 242 retrospectively collected FFPE CRC samples was tested with the NRAS-BRAF mutation test. Results were compared with assessments made by the MassARRAY comparator reference test. Of the 242 FFPE samples analysed by the NRAS-BRAF Mutation Test, 36 tested positive for a mutation in NRAS, and in 205 cases, no NRAS mutation was found (table 3). As presented in table 4, results obtained by the NRAS-BRAF mutation test for NRAS testing were in agreement with the results obtained from the MassARRAY comparator reference test for 241 of 242 samples, resulting in an overall concordance of 99.59%, (lower bound of one-sided 95% CI=98.17%) with positive agreement of 100% (lower bound of one-sided 95% CI=93.0%) and a negative agreement of 99.51% (lower bound of one-sided 95% CI=97.85%).

Table 3

Mutation concordance analysis for the Idylla NRAS-BRAF mutation test for NRAS compared with MassARRAY. Table entries on the diagonal with orange shading are concordant results. The one discordant result, G12D with Idylla and WT with MassARRAY, is shaded light blue and discussed in the text.

Table 4

Idylla NRAS-BRAF mutation test results compared with the reference test (MassARRAY) results

Of the 242 FFPE samples, 52 tested positive for BRAF V600E/D mutation; no mutation was found in 190 cases. Results obtained by the NRAS-BRAF mutation test for BRAF testing were fully in agreement with the results obtained from the MassARRAY comparator reference test resulting in an overall concordance of 100% (lower bound of one-sided 95% CI=98.89%) (table 4).

Discordant result

In one case, a G12D NRAS mutation was detected by the NRAS-BRAF mutation test but not by the MassARRAY comparator reference test (tables 3 and 4). The NRAS G12D result was confirmed to be a true positive by ddPCR analysis with an allelic frequency of around 0.5%, which is well below the 5% limit of detection of the MassARRAY system (table 5).

Table 5

Discordant result between Idylla NRAS-BRAF mutation test and MassARRAY reference method


The importance of testing for NRAS and BRAF mutations in tumours from patients with CRC is reflected in recent clinical practice recommendations of the American Society of Clinical Oncology27 and the European Society for Medical Oncology.28 NRAS mutations in colorectal tumours are associated with poor response to anti-EGFR therapy while BRAF mutations are associated with reduced survival.29 Evidence-based guidelines therefore recommend NRAS (predictive) testing to guide treatment decisions and BRAF (prognostic) testing to provide information on outcomes.27 28

The performance of the Idylla platform has been previously reported in various validation studies.23 24 30–33 In the present study, results from the Idylla NRAS-BRAF mutation test, which can detect 18 NRAS and five BRAF clinically relevant and actionable mutation targets, were compared with results obtained from NRAS and BRAF mutation testing by the MassARRAY.

There was greater than 99.5% concordance between the Idylla platform results and MassArray findings for NRAS and BRAF testing. Droplet digital PCR testing of the one discordant result confirmed the Idylla identification. One plausible explanation for this finding is that Idylla may have a lower limit of detection than the MassARRAY reference test; other possible explanations include test error and sample error.

Both the Idylla platform and MassARRAY methods were highly sensitive, and mutations could be reliably detected in small core biopsies and in samples with extensive mucinosis or necrosis.

As with other products based on similar technologies that identify specific mutations, mutations that are not intended to be detected will not be detected. However, these technologies in general, and the Idylla system, in particular, have the advantage of being able to readily update their assays to include mutations newly recognised as being clinically important, although revalidation for CE marking may be required.

In conclusion, the Idylla NRAS-BRAF mutation test is a fully automated and integrated real-time PCR-based test with high sensitivity and rapid turnaround time, providing results that are readily interpreted by clinicians. The Idylla platform proved straightforward to use with or without macrodissection. The method of handling the FFPE sections is distinct to Idylla and contrasts with the more usual method of placing the FFPE sections into a container. FFPE samples are loaded directly into single use cartridges with minimal sample preparation: histological sections (curls) for analysis are easily ‘sandwiched’ between the filter papers in the cassette. Potential drawbacks include the limited throughput (four samples per unit per day) and inability to retrieve samples and rescue an analysis if the system malfunctions mid-run. The Idylla platform would ideally be set up in a clean room close to where the paraffin sections are being prepared in a pathology laboratory and employed in clinical settings where the time saved by analysis on demand (compared with routine batch processing) would provide clinical and/or efficiency benefits. The Agena MassARRAY system can test from 1 to 384 samples on a chip (depending on the panels of tests and number of wells) and can process around 120 colorectal mutation tests per day.

Take home messages

  • Identification of mutations in NRAS and BRAF is important for treatment and prognosis in patients with colorectal cancer.

  • The Idylla NRAS-BRAF mutation test is as accurate as the widely used Agena MassARRAY.

  • The Idylla platform and Agena MassARRAY system are both highly configurable for low to high throughputs, but address different clinical and processing needs. The Idylla platform provides rapid analysis on demand, thus allowing pathology reports to integrate histology and mutation analysis, and treatment decisions to be made with minimum delay. The Agena MassARRAY system can process around 120 colorectal mutation tests per day.


The authors thank Medical Research Council/Engineering and Physical Sciences Research Council Newcastle Molecular Pathology Node for providing the support for this study. We also thank the reviewers for their constructive comments.


View Abstract


  • Contributors Biocartis devised the project; LJ managed the project; TS and TM in Aarhus and AL in Newcastle located the CRC samples; Idylla and MassARRAY testing was carried out by NewGene ; LJ and MP performed the analyses; all authors contributed to the final manuscript.

  • Funding The Idylla system and cartridges were provided by Biocartis who funded the study.

  • Competing interests EV is employed by Biocartis. Agena Bioscience have provided training, assay reagents and supported conference attendance by NewGene staff (AS, BC, AJL, LL) during the 36 months prior to this publication. All other authors have no completing interest to decline.

  • Ethics approval National Research and Ethics Service (London—Bloomsbury Research and Ethics Committee, reference 16/LO/0509) and the Central Denmark Region Committees on Health Research Ethics.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement No additional unpublished data are available for sharing.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.