Blue Cloud Ventures recently hosted a roundtable event with 11 companies in our network to share insights on how they are implementing LLMs in their organizations. In the initial installment of our two-part series, we explore common challenges surfaced during our discussion.
The impressive capabilities of Large Language Models (LLMs) hold great promise for B2B SaaS companies looking to enhance internal productivity or elevate their product offerings. Many of these companies have acknowledged this opportunity and are actively incorporating LLMs in their organizations today.
Yet while LLMs could be transformative, we recognize that the road to successful implementation is not without its hurdles. Turning these technologies into commercially viable products or leveraging them for internal operations can be a challenging undertaking.
Blue Cloud Ventures recently co-hosted an event with Jeavio, bringing together thought leaders from a variety of companies within our network, including many of our own portfolio companies. Participants shared unique insights on challenges, best practices, use cases, and infrastructure/deployment considerations when launching products and features powered by LLMs.
In the initial installment of our two-part series, we explore the key challenges our roundtable participants grappled with during the process of incorporating LLMs into their respective organizations.
-
Cost: Implementing and using LLMs can be expensive for organizations due to several factors:
a. Inference: Running the model to make predictions or generate outputs may require significant computational resources. If an organization is using the model frequently, or if it is serving many users, inference costs can quickly add up.
b. Fine-tuning: Tweaking the model to perform better on specific tasks also requires meaningful compute.
c. Data: LLMs require large amounts of high-quality data for training. Gathering, cleaning, and maintaining these data can be a costly process. - Accuracy: Deploying LLMs for Q&A use cases for instance, could become a challenging balancing act between deterministic and probabilistic responses. This balance often requires additional rounds of fine-tuning and training before the technology reaches proficiency levels useful for specific, domain-focused applications.
- Performance: The performance of LLMs presents another layer of complexity. With inherent constraints due to limited context windows, these models can often be slow in inference. Some participants discussed the use of sparser models and distillation techniques to improve inference performance.
- IP Concerns: Intellectual Property issues remain a persistent concern, especially regarding data provenance and code generation using models operating on restrictively licensed code.
- Model Types: When it comes to choosing between specialized and general-purpose models, the decision is rarely straightforward. Several participants highlighted the need for deep integrations with existing technology stacks to make LLMs truly useful, considering their contextual limitations.
- Security: Security considerations cannot be brushed aside. Risks such as prompt injection, the potential generation of insecure code, and the possibility of internal data leakages to external entities present significant security challenges. Some organizations are circumventing these risks by internally using models like GPT-4 through an API, and incorporating additional security measures, such as the removal of personally identifiable information (PII) early on.
Conclusion
While the potential of LLMs continues to generate significant excitement within our community, implementing LLMs in organizations is no small feat. It involves a complex range of considerations from performance and cost optimization to ensuring security and data privacy.
Certain obstacles such as sub-optimal performance can be remedied through optimizing pipelines, sparsification, and other techniques, yet others like IP concerns and the unpredictable nature of LLM behavior present more significant barriers to their universal adoption.
Stay tuned for the next installment of this two-part series where we will dive deeper into some of the key considerations highlighted during our discussion. We will also shed light on some of the tools, frameworks, and methods organizations are employing to tackle these challenges.
We welcome you to join the dialogue and encourage you to reach out with any questions or thoughts you may have. Please feel free to contact us at operations@bluecloudventures.com.
Victor is the Head of Europe for Blue Cloud Ventures, leading the firm’s efforts across the continent. He has been involved with BCV’s investments in Paradox, Trulioo, Go1, Cognism, Templafy, and LMS365, where he is active at the board level. He focuses on security, HR tech, sales tech, and business applications. His previous experience includes roles at BNP Paribas, Compass Group, and Ellington.