1.2 Evaluating AI Tools for Grant Writing Tasks

Learning Objective

By the end of this lesson, you will have a sense of what AI tools exist and be able to select appropriate tools depending on the grant writing task.

Types of AI Tools

AI tools that can be used for grant writing fall into several categories, each with different strengths and use cases. Understanding these distinctions can help you find the right tool for each proposal-related task.

Text-Generation

Large Language Models (LLMs)

LLM definition [1]

Large language models (LLMs) are advanced AI systems that understand and generate natural language, or human-like text, using the data they’ve been trained on through ~[machine learning](https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-machine-learning-platform/)~ techniques. LLMs can automatically generate text-based content, which can be applied to a myriad of uses cases across industries, resulting in greater efficiencies and cost savings for organizations worldwide.

LLMs, including ChatGPTClaude, and Gemini, are among the most versatile AI tools and can be used for several grant-writing-related tasks.[2] These models excel at:

  • Generating initial drafts of proposal sections.

  • Assisting with edits and revisions to improve clarity, tone, and structure.

  • Conducting research by summarizing information from multiple sources.

  • Reformatting content to match proposal requirements.

Their strengths include versatility and the ability to understand context (when given the right prompts). They can also handle complex instructions. However, LLMs also have downsides. They may generate false information and can be unreliable in areas requiring specialized knowledge. Because of these risks, all LLM-generated content must be fact-checked.

In the grant writing context, LLMs are best for preparing initial drafts of proposal narrative sections, needs assessments, or program descriptions.  

Research

For research, research-focused AI platforms like Perplexity and Elicit  can be better options than a general tool like Microsoft Copilot, which works well for more general research questions. AI tools like Perplexity and Elicit can help with several common research tasks, such as:

  • Source identification: Finding relevant research papers, reports, and data by reviewing online sources (Perplexity).

  • Literature reviews: Summarizing key findings across multiple studies described in peer-reviewed articles (Elicit).

  • Citation management: Organizing and formatting references (Perplexity).

  • Fact verification: Cross-checking claims against multiple sources (Perplexity for web content, Elicit for academic sources).

Using a research-focused tool has several benefits. The source attribution tends to be stronger, making it ideal for specialized research tasks. However, like all AI tools, research-focused AI tools only have access to public sources, so content in proprietary (i.e., paywalled) databases will not appear.

In the grant writing context, tools like Elicit are best for tasks such as background research, literature reviews, data collection, and evidence gathering.

Tool Selection Criteria

When choosing between AI tools, consider the following:

Task Complexity

Which AI tool will be most appropriate depends on the task's complexity.

  • Simple tasks: For a routine task, general tools like ChatGPT or Microsoft Copilot, should be fine.

  • Complex tasks: For background research, a more specialized tool like Elicit or another advanced LLM will probably be the better choice.

Accuracy Requirements

While all AI output must be fact-checked to confirm its accuracy, some models are more accurate than others.

  • First-draft content: Basic models, such as ChatGPT or MS Copilot, are fine for generating "starter text" that can serve as a basis for a first draft.

  • Technical content that requires higher levels of accuracy is better with a research-focused tool that offers source verification features, such as Perplexity.

Time Constraints

Another factor to consider when choosing a tool is your familiarity with different AI platforms.

  • Immediate deadline: If you are facing an urgent deadline, you don't have time to learn a new tool, so pick the one you are most familiar with and find easiest to use.

  • Distant deadline: If you are working on a project with a longer time horizon, choose the best tool for the task, which might be a specialized tool.

Other Factors

  • Budget considerations: If paid tools are not an option for the entire organization due to cost, one option is to offer a paid version of the preferred tool to staff who require access to premium features. Another option is to stick to tools that offer free plans, though these typically have security and usage limits that can significantly limit a tool’s utility.

  • Technical expertise: Introducing an AI tool and training staff on it can exceed the resources of smaller organizations. For smaller organizations, free tools may be the best option, although staff training will still be essential.

  • Security requirements: Using any third-party software, including AI tools, raises concerns about data privacy and confidentiality. Data and security policies vary among AI tools. Particularly for organizations opting for a free tool, the privacy terms and settings must be carefully evaluated to ensure maximum protection of confidential or proprietary information.

Practical Evaluation Framework

To evaluate AI tools for your needs, work through this checklist:

  • Define your specific use case(s): What do you need help doing?

  • Test tools with real-world tasks: Test the quality of a tool’s output by uploading a section of a recent proposal and asking it to draft similar text for a different opportunity (note: you will need to provide the AI tool with the relevant solicitation guidelines). You can conduct similar tests to evaluate how the tool performs with everyday tasks, such as generating tables and summarizing background research.

  • Evaluate output accuracy, relevance, and usefulness: In addition to comparing how a tool does with text generation, also evaluate a tool’s output based on objective factors, such as accuracy, and more subjective ones, including overall quality and responsiveness to the solicitation.

  • Consider integration needs: How well does the AI tool align with your existing processes? Additionally, decide whether you want your conversation (chat) history preserved. Not all tools do this, especially in their free versions.

  • Calculate resource investment: In addition to licensing costs, consider the time investment required to master and integrate the tool into existing workflows.

Suggested Tool Combinations

If your work situation permits, consider creating a shortlist of tools for different purposes instead of committing to a single tool. For example:

  • Research phase: For conducting background research in preparation for writing a proposal, a specialized tool such as Perplexity is one option. Another option for general research is Microsoft Copilot, which also reliably adds source notes to its output.

  • Writing phase: For the proposal writing phase, a general-purpose LLM like Claude, ChatGPT, or Google Gemini can be effective. Microsoft Copilot is another good choice if you work within the Microsoft ecosystem.

  • Review phase: Document analysis tools for compliance and consistency. General-purpose LLMs can be used for this task. However, because this usually requires uploading proposal text, it is advisable to use an enterprise (paid) version of a tool for the review process because they offer greater security.

  • Final editing: For editing, writing-focused tools such as Grammarly are an option, as well as the grammar and editing checking features built into Microsoft Word and Google Docs.

Lesson Assignment

Tool Evaluation Exercise:

  1. Identify a specific grant writing task that you perform regularly (e.g., writing background and context sections, developing evaluation plans, creating budget narratives)

  2. Test the same task using two different AI tools:

    • One general text generation model (e.g., ChatGPT or Claude)

    • One specialized tool or research platform (e.g., Perplexity or Elicit)

  3. Document your comparison:

    • Which tool produced more useful initial output?

    • Which required less editing and revision?

    • Which better understood the context and requirements?

    • Which would you choose for this task in the future?

Coming Up Next

In Lesson 1.3, we'll learn how to write effective prompts that generate useful content for grant proposals, starting with basic prompting principles and moving to advanced techniques.


[1] Source: Microsoft, https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-are-large-language-models-llms

[2] Microsoft Copilot is not on this list because it is an AI system powered by OpenAI’s ChatGPT LLM models that works in conjunction with Microsoft applications, such as Word and Excel. Microsoft 365 Copilot, https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-overview


Lesson Summary