Some basic conversation about LLM performance metrics

Lately open source large language models proliferate like the bunny rabbits in my yard and worse than that is the specter you might have to find a good reason to pick one over another.

In this video, I recommend HuggingFace's LLM leaderboard as a place to get started. It ranks these models according to a number of metrics with the widely-discussed Falcon presently leading in average score. I will drill into these metrics a bit... Just what do they measure and how? How might relative performance be different in the applications we are pursuing at my company? One point I would like to keep re-emphasizing is that the breadth and flexibility that might have attracted you to OpenAI's ChatGPT will not only be one of the more difficult qualities to reproduce in house, but it is also likely to be one of the more difficult qualities to understand via a standardized test.

Want your own LLM? Some thoughts on the ins and outs of open source options

Does $200k and 9.5 days to train a model sound cheap and fast to you? This is a serious question, and you should also consider the odds that your first shot hits the desired target.

In this video, I begin my promised discussion of open source large language models. If you are an organization that wants to build its own LLM, refining an open source model looks like the right path to me at the moment. However, I see a lot of danger that it could be more expensive than some people think, more poorly performing, and leave you with some of the problems that scared you away from refining a model like OpenAI's celebrated ChatGPT. If nothing else, you will have a lot of decisions to make and giving you context for those decisions is what I hope to do here.

If you are not careful, LLM will mention exactly what you tell it not to

If you say "Don't mention topic X" a human adult will usually oblige you but a toddler or a large language model will often turn around and mention it immediately.

It is a surprisingly tough to keep a generative AI tool like ChatGPT from NOT doing a particular thing and often a ham-handed attempt will actually solicit the forbidden action. In this video, I give an example prompt that demonstrates this unfortunate property and hopefully also manage to give you a little prompt engineering intuition about why things work out this way.

It's hard to tell artificial intelligence NOT to do something

Do you ever fill a closet with freshly poured glasses of beer? I don't think many people do, but it may be impossible to find an image for your generative AI widget's training data that embodies a particular strange thing not happening.

In this video, I examine an example of an artificially generated image that went a little bit wrong in an interesting and subtle way. I use it to explore some themes around prompt engineering, notably that it is problematically difficult at times to give negative (DON'T do this) instructions and also that it is important to remember the ways in which artificial intelligence does not really understand what it is doing.

Why LLM is big, shadow IT, and what is realistic in-house

It is undersold that tools like ChatGPT, when compared with some of the larger universe of things we might call artificial intelligence, is very easy to get started using with no special expertise and help from other data and IT systems at a business. This is probably enough to make it a born shadow IT classic, but it also appears to me at this (certainly fleeting and soon to evolve) moment that efforts to provide similar in-house tools to compete with the shadow-IT-on-phone option could turn out to be complicated and ultimately disappointing in performance.

Yet another OpenAI lawsuit, this one around novels and copyright

Two authors have filed suit against OpenAI for (allegedly) wrongly using their novels as training data, deciding to do so after noticing that ChatGPT could summarize their novels in great detail. This lawsuit notably involves copyright protection and not some other areas of intellectual property law that are already being litigated. I do cover some of the details of this lawsuit, what merits it might or might not have, and how these issues intersect with the realities of trying to train a large language model. However, as someone who has posted a few videos about OpenAI's legal adventures the better takeaway might be that the law around important practices here is unsettled, some of the best and most accessible LLMs are presently more than a little sketchy legally, and this is something to watch and manage if you want to exploit these tools in your business.

OpenAI targeted in $3 billion class-action lawsuit

OpenAI and benefactor Microsoft, maker of ChatGPT, are now the target of a $3 billion class action lawsuit alleging that they broke the law in scraping the web for training data. The merits of the lawsuit are not fully clear to me, but I can say that massive scraping is probably necessary for a large language model with ChatGPT's flexibility, various fights about this scraping (including this one and beyond it) are already heating up, and web scraping is in fact a legally murky activity.

Update: Ghost PII in iOS

My colleague Jack Phillips and I have an update on our demonstration app built for iOS via Swift. The point, of course, is to show what Ghost PII is about, how it can be used to keep data encrypted everywhere but the user's phone including while in use in computations on the server, and how it is easily layered on top of other technologies. I would say we have had unique success already making this kind of technology accessible and easy-to-use and the ultimate goal of this project is a library that will provide iOS developers the value without requiring them to even know Ghost PII is there.

Mega Breach at MOVEit Managed Data Transfer

It felt irresponsible to wait too long to let you know about the HUGE data breach over the weekend involving the MOVEit Managed File Transfer tool. Short story: it can be hard to send people big datasets, maybe you decide to pay someone to use a tool for this, and if there is a cyber security bug in the tool everybody is really screwed and multiple government agencies are giving up the juiciest of PII on basically the whole population of Louisiana.

An unusual unionization drive at Alphabet with AI intersections

I couldn't let the the recent unionization drive among Alphabet contract workers sneak into the weekend - it might be a great place to read tea leaves for clues on the future of generative AI and large language models specifically.

These models, thus far, have required significant amounts of skilled human labor to train and refine. For this and other reasons it is a pretty expensive to be in and you can guess some things about how many and how big are the companies that will be selling you these things in the foreseeable future.

DeSantis deepfakes Trump! An unfortunate milestone...

It appears the DeSantis campaign has circulated a deepfake (a fake image generated using artificial intelligence techniques) of Trump warmly embracing Fauci (who is very unpopular with GOP base voters) in what appears to be a smear for political advantage. This is a notable "first" that I think many of us knew was coming and we may soon begin learn a lot about the kinetics of deepfake propaganda from events in the wild. This video discusses the basic facts and what to watch for... and I recommend you watch.

Interest rates as a cultural force

You're about to get tired of Marvel because the Federal Reserve is no longer maintaining its post 2008 crash ZIRP (zero interest rate policy)...

This is (probably) crazy talk. It is true, though, that interest rates influence culture in subtle but real ways and the recent shift in the macroeconomic climate is likely to create a shift in culture as well. In this video, I talk some basics of recent monetary policy, the everyday "meaning" of interest rates, and theorize more seriously that people like Elon Musk who are pining for the office hustle culture of a past era may be pining a long time for something that can only grow in a very particular sort of soil.

An AI nightmare at the National Eating Disorder Association

This video covers what one can only call an artificial intelligence chat bot adoption horror story at the National Eating Disorder Alliance. Spoiler: the help line is a robot and its giving optimally terrible advice. Hype is determined by the cost-benefit calculus of posting on social media. The costs and benefits for any given application in a business may be in very different balance, and sometimes it's hard to say just what these new models are likely to do in the wild.

Gossip on the biometric beer stand at Coors Field

You may have seen articles on LinkedIn News last week about Amazon piloting a system by which stadium-goers can pay for beer via a palm print biometric. As it happens, my colleague Jack Phillips was at Coors Field to see the Colorado Rockies over the weekend and here he kindly shares the word on the street about this new payment technology.

In the background here is that unusual data privacy concerns surround biometric data like palm prints and this data is tightly regulated in many states.

Updates on our upcoming iOS tools and future-proofing data sovereignty

We've been working on some powerful iOS developer tools powered by Ghost PII. I open by talking through why you might want to do this: I posted a video yesterday on Meta’s headaches moving data from the European Union to the United States and this sort of a technology is a way to avoid this problem by making sure that data is never in plaintext anywhere but on the end user's computer.

My colleague Jack Phillips will help you peek a bit under the hood at how all of this works, including showing just what (securely encrypted data) will actually be stored in Firestore (just what we happen to be using in this case) and teasing a bit about the next phase in which we build some analytics that work on this data directly without need of encryption.

The European Data Protection Board announced a record $1.3 billion fine on Meta, parent of Facebook and Instagram, for illegally transferring data on European Union nationals to the United States. In addition to the basic facts, I review the abundant and important backstory for these events going all the way back to Edward Snowden's surveillance revelations, the history of data privacy frameworks going down in flames following lawsuits by Max Schrems, and the complicated political intersections that drag this all out forever including the unique role played by regulators in Ireland.

Juicy tidbits from the history and philosophy of AI

The history and philosophy of artificial intelligence has some surprisingly useful nuggets in it that orbit the question: Is human intelligence "real"? What is it?

I talk about the two previous cycles of AI hype and collapse into "AI winter" first in the 1960s and then in the 1980s. In the former, academic philosophers like Hubert Dreyfus brought interesting critique orbiting around issues like misunderstanding of human consciousness (in a sense still our only available prototype for AI) and how difficult it is to separate what we understand as intelligence in ourselves from the visceral desires that come from our body. I would say these criticisms anticipated the use of reinforcement learning, among other things, and they might have more practicality in them than you might think.

Fear marketing in artificial intelligence

Let's peer behind the veil at the marketing aspects of artificial intelligence (notably LLMs like ChatGPT of late) and observe how much of the psychology of fear is in play. Is this something we should snap out of a bit? Imo, yes.

I discuss how the concerns of Geoffrey Hinton, the reported "Godfather of AI" who left Google to talk about the dangers of AI, actually differ very little from the public statements of top executives at companies like Microsoft and his former employer. What gives? It appears fear is great marketing for AI, and I go into further detail on why the widely discussed potential pause on AI research is probably mostly conscious marketing by the public figures discussing it.