White box

Commonly used by organisations after a black box pentest to validate controls, or as an assurance to the business for audit purposes. A white box pentest is prescriptive in nature and is the only type of pentest that provides an attestation of compliance.

What to provide

Preparing for an effective whitebox pentest is key to it's success.

• Formal engagement agreement
• Reduction Analysis
• System Architecture Diagram (SAD)
• GNS3 validated network topology
• Whitelist Inventory (complete list of domains, software, and dependencies)
• Blacklist Inventory

Optionally provide

• Credentials: 2 or more per each RBAC or ABAC or other access levels
• Dictionary of terms, unique to your proprietary business, sector, or partners
• Source code (often in place of Inventory)

The quality of findings and coverage are directly proportionate to the level of detail disclosed to the tester.

Redundant Efforts

• Open Source Intelligence (OSINT) gathering
• Remote System Discovery
• Social Engineering (obtain credentials, physical access, etc)

Advantages

The organisation defines the attack surface which can be appropriate for various reasons such as fragile systems' availability, Material data integrity, or maintain confidentiality of trade secrets.

The level of insight allows testers to better understand intended use to find and exploit undesirable side effects the organisation may be unaware of. This also translates into concise recommendations for remediation and controls.

Automation of white box tests can also be achieved to avoid regression into the future and improve coverage and effectiveness of future testing efforts.

Completed on a time-cost basis appropriate to your budget and risk appetite, as these tests represent an infinite ongoing opportunity you chose the amount of effort you can afford or need.

A complete white box test report provides a timed attestation on the security posture of your organisation for the scope of test procedures performed. A good tester provides both a public appropriate report that you might distribute to customers and one that is comprehensive and useful for your security strategy and auditors.

Disadvantages

Testers often find themselves having a much higher barrier to entry for any given white box test as they are expected to have an intimate knowledge of the systems, beyond the developers of the systems themselves. This limits the pentesting organisation to use only resources with the experience and ability unique to the target rather than the experience of only being a pentester.

Due to the scope and complexity of systems some conditions will initially be left untested, it is the goal of continued testing, and the automation of prior testing, to reach a broader coverage of conditions tested over time.

The prescriptive nature of this testing excludes the tester to explore any unknown unknowns, meaning the tests are performed based on the functionality that exists at the time, and is known through the disclosure of the target to the tester.

The targeted nature of these tests can cause system availability, integrity, or confidentiality issues. Ensure that the testers are capable of providing detailed tracing reports helping you identify and validate the effects of the tests performed. Therefore, ensure your BCP and DR strategy is tested and validated, internal and external parties are aware in advance of possible disruptions, and you do a complete audit of system integrity before assuming a BAU posture.

Black box

Often described as requirements based or specification based testing, where an ethical attacker simulates an external malicious attacker with no prior knowledge of the targeted systems.

What to provide

• Formal engagement agreement defining the rules of engagement (RoE)
• Copy of the legal permit to conduct the tests from regulatory, prudential, compliance, or any other relevant obligations such as contractual with service providers and downstream partners
• Upfront status report and meeting expectations

Advantages

Organisations with trade secrets and patents (or any other legal expectation of secrets) often find the black box tests extremely useful to find public sources of confidential information and better understand the threats to their secrets. This may lead to legal action to remove or control any such findings, however the intention of the tester is to find and demonstrate the threats only.

While internal systems (including those that transit the world wide internet) are likely being monitored already, it is scrutiny of the wider public internet that a tester performs that can give you additional ongoing monitoring capabilities. Such as monitoring pastebin or other code dumping repositories, identifying things you weren't aware of that are actually internal to your business such as employee social media with company branding, public 3rd party tools used by employees, directories that aggregate your brand in some way.

Disadvantages

Organisations feel a false-sense-of-security that the path the ethical attacker took is the same paths available to malicious attackers - this is fundamentally untrue as the ethical attacker abides by an RoE and applicable laws whereas the true threats lie outside these boundaries.

The findings report shows you where identified vulnerabilities exist, it is not comprehensive to be sufficient for your security strategy alone, and it is not collateral useful for public disclosure or to be useful for auditors (it is not considered an attestation of any kind)

This type of testing is highly likely to cause availability, integrity, or confidentiality issues. Therefore ensure your BCP and DR strategy is tested and validated, internal and external parties are aware in advance of expected disruptions, and you do a complete audit of system integrity before assuming a BAU posture

Grey box

The most common tests performed, often misunderstood by organisations as being one of either the white or black box types after all the engagement requirements have been discussed and decided. While grey box testing gives you the best of both worlds on the surface, it is actually the opposite as it makes most advantages of both cancel the other out.

You may wish the testers to simulate scenarios such as a malicious insider threat where specific information is disclosed to a competitor so as to identify the risks to the business.

What to provide

The reason grey box testing exists is that it combines any variation of white and black box testing.

It is common that successful grey box testing is done with a defined scenario, otherwise testers will be burdened with disparate and conflicting goals.

Advantages

Grey box scenario can be extremely useful to identify various insider threats, such as system misuse (accidental or malicious), information disclosure, rouge technology, and leaked credentials.

Findings can also help decide on various educational initiatives such as senior staff or executives conducting themselves in a high risk manner.

Detect the existence of 'godmode'. This is where an employee with legitimate access to perform high risk actions does so for reasons outside tier role or not in the business interest. For example a manager giving another manager access to bypass controls that are inconvenient.

Disadvantages

A grey box test report may include the same structure and depth of content as a white box test report however it can not be used as an attestation as the requirements were not disclosed up front.

Insider knowledge of the grey box testing procedures negates the threat scenarios that simulate external malicious attackers.

Conclusion

The unfortunate reality is most organisations commission either (or both) the white or black box tests and get neither, instead the tests better represent a grey box test. This can be due to the maturity of the testing organisation, or a lack of knowledge on the side of the target organisation when defining requirements. To avoid falling into grey box unintentionally an organisation should first understand their requirements and split efforts of black, white, and grey box testing across distinct testing organisations, and in future engagements avoid using the same testing organisation for a different test type in case they reuse knowledge from past tests. This will encourage the testers to perform tasks within expected boundaries, and provide assurance to the organisation and auditors that tests are performed without bias and within the defined requirements.