Anchor Oracle Price Discrepancy on December 9th

[PFC]

At 08:43:37 PM on December 9th (UTC), the Anchor Oracle Price Feeder registered a $bLUNA price of $58.44, significantly lower than the reported $LUNA price on exogenous exchange venues, quoted as ~$66.10 on Binance’s LUNA/USDT pair at the time.

The inaccurate price reported by the Anchor Oracle Price Feeder caused erroneous liquidations for 239 users who had open borrow positions below the max LTV of 60%. In total, ~690,727.49 bLUNA (~$37.249M total discounted value) were liquidated. Two additional price discrepancies occurred subsequently, approximately 30 mins apart from the original event. These events triggered 3 total liquidations. However, the bulk of affected liquidations occurred immediately after the initial price discrepancy.

We’re currently investigating the precise source of the issue, and in the meantime, have implemented a patch fix for the Anchor Oracle Price Feeder to mitigate any further discrepancies.

Of note, we have an impending governance proposal to migrate the Anchor Oracle Price Feeder and replace it with a combination of the following:

bLUNA/LUNA * Chainlink – to use the primary source of price of bLUNA from the Chainlink price oracles

The new price feed structure sources prices from Chainlink’s venerated oracle network + the Anchor Hub Contract, which precisely tracks the LUNA/bLUNA ratio since all bLUNA minting passes through it.

We will release more details about the precise mechanics of the issue once we’ve concluded our investigation into the error. Concerning the affected users, there are several important notes and remedial steps to consider.

First, a subset of the 242 affected borrowers on Anchor had open positions for both bLUNA & bETH – totaling 102.74 bETH ($0.359M total discounted value at trigger price). These users were liquidated for both bLUNA and bETH since liquidation is proportional.

That part is notable as many users claimed bETH was also affected by the oracle feed mishap. However, bETH liquidations for those users were a consequence of also having a bLUNA position open – which was the only collateral affected by the faulty oracle price.

We’re currently querying the individual borrower state by block height for the event, but the general calculations of the event are below. Using the average price from the bLUNA oracle of ~66.35 for the affected collateral immediately before and after the initial price discrepancy, we can calculate the following.

bETH value + bLUNA value = $46,247,494.37

Reimbursing any liquidation premium and price discrepancy gap for the faulty price would take every liquidated position back to neutral. For example:

Total value liquidated - Sold off liquidation = Amount to reimburse.

The total to reimburse affected users:

bLUNA repay = 37,249,938.26

bETH repay = 359,251.19

Total repay = 37,249,938.26 + 359,251.19 = $37,609,189.45

To reimburse: 46,247,494.37- 37,609,189.45 = $8,638,304.92

Including the liquidations from the following 2 oracle price discrepancies, which are smaller, the final total will likely be closer to $9 million. We’ll have a more precise figure once we’re finished calculating per borrower contract balances at different block heights.

We encourage the community to create a governance proposal to compensate the affected users of this oracle pricing issue via $ANC allocations from the community pool, where we can discuss openly in the comments below.

Other options may be available and we encourage the community to explore them openly. Please direct your feedback to the comments in this forum post.

Ian/PFC

[paletas]

Let me start off by saying I’m 100% in favor of compensating affected users, I was not liquidated yet I can only imagine the feeling of having been by a faulty oracle and this compensation is a step of good faith from the protocol.

There’s two major ways to do it, imo:

1. Use the $ANC allocations from the community pool as suggested;
2. Use funds from the yield reserve, at least I believe that to be a viable option;

Option 1 will lead to a massive sell off on the $ANC token and that’s a major hit on stakers, which is to be expected as a staker myself, but kind of disheartening if I’m to be completly transparent. Option 2 will put the protocol in a though spot, losing liquidity that’s there to ensure the yield that’s already taking an hit from market conditions.

I believe we should consider both options in a mix, 50-50 or some other combination, as to avoid a $9M dump on the market and soften the blow a little bit.

[codeNoodle]

I agree that we should definitely vote to reimburse the affected users.

To add on to point #1 by paletas above, if we reimburse 100% with ANC tokens then it is very likely to lead to the selling of at least a majority of those tokens. This could change the number of “affected” users from 242 to a much larger number including all ANC stakers/LPs.

I believe that a mix of both ANC tokens and UST from yield reserve would be the best solution in this case. As for the ratio of amounts, I still think it should be at least 50% ANC tokens. In the end, I would be fine with 100% ANC tokens as well, but obviously lessening the selling pressure would be ideal I’d think.

I was not liquidated myself, so would love to hear from some of those individuals’ perspectives if possible! Disclaimer: I stake and LP ANC.

[wengzilla]

Thanks for this thorough analysis! Really hopeful that the community bands together to help compensate affected users.

I think tapping into the ANC reserves or the yield reserves are both reasonable options. Alternatively, is it possible to use some of the UST allocated for Ozone reserves ($4B+) to pay for this? In my opinion, this is potentially the best option as it leaves the $ANC price intact, allows for a robust yield reserve, and would barely affect Ozone reserves.

[paletas]

Problem I have with the Ozone reserves is that it’s a TFL fund, not an Anchor fund, even if Anchor was founded by TFL it should strive to live on it’s own, not running back to mom and dad every time something goes wrong. I’d rather we solve this in-house and prove the protocol can stand on it’s own two legs, without the backing of TFL.

[codehans]

So the specific drop was from block 5613386 to block 5613387,

Anchor’s price oracle reporting (respectively) 66.413162312092379615 and then 58.444236122666136581.

I think it might be useful for the community to know how the Terra Validator price voting protocol changed, as there was a similar swing there, 3 blocks earlier: the Terra Oracle reported the Luna price at block 5613383 to be 66.412685092457964386 and at block 5613384 it was 58.443816164693572794.

Note that these are not the same as the Anchor Oracle reports, but have the same swing. Potentially indicative that the Terra Oracle feeds the Anchor Oracle.

This - in my personal opinion - looks more like a failing of the Terra Oracle than it does specific to Anchor, and that Anchor in fact have been (thankfully) the only protocol affected by this. It’s great to see this response to an event like this, although perhaps this is worth the community considering before looking to Anchor to shoulder the full burden of remuneration.

[wengzilla]

I don’t disagree entirely, however, TFL (or Anchor?) has been operating an extremely profitable liquidator (and I believe continues to place bids on Kujira)… Feels like it’s not a great look to be profiting off the bug in the first place.

[paletas]

That’s a good catch, I believe there was an update made to validators indicating that the problem could have come from faulty terra oracles. If that’s the case and it’s proven to be a network wide problem, and should TFL choose to contribute to the restitution of the Anchor users, then I’d have no issue accepting that as a solution.

[paletas]

That’s a fair point and something I failed to consider with my answer.

[codehans]

v0.5.13 of Core was released today, contents unknown and its update pushed through quite strongly, particularly for oracle validators. This seems like a (very efficient and appropriate) response to a zero-day exploit, and at the minimum balances out the case for remuneration - if there is one at all.

FWIW I have no strong opinions either way, I’m just here to present observations as I’ve seen them

[somethingelse]

If we go this route of using the community funds to pay to reimburse, perhaps we should designate trusted parties with a multisig wallet to sell the ANC for UST in a transparent manner, very similar to how the Terra community pool burn of LUNA to UST was performed. This way, there is not an immediate dump but a controlled, and communicated selling of ANC, giving dip buyers the opportunity to know exactly when to make their move.

[ColonelSanders]

It makes perfect sense to use the community pool to reimburse people that were liquidated. Yes, it should come at the expensed of ANC holders as this increases ANC supply/increases selling pressure on ANC. This is an Anchor problem and ANC holders alone should bear the cost of this. Drawing down from the yield reserve isn’t the right answer here. The yield reserve needs to still be beefed up significantly in order to get to the level that it will be able to weather any and all storms in the future.

Is there really a debate here on whether to use ANC community funds or the yield reserve? It seems pretty clear cut to me.

[planktonfarmer]

First of all, great job by the team. I believe the fix was available within hours of it being reported, no doubt preventing further losses. This could have been significantly worse and that rapid response should be applauded. Well done to everyone that worked on it.

Do we know how much profit the that liquidator bot made over it’s course? TFL had a monopoly on liquidations during that time so I imagine it to be substantial.

I think, if TFL showed some good faith in offering some of their (stupendously deep) coffers, whether from the bot or not, it would only be fair for ANC holders to cop up as well. A 50/50 split seems reasonable. I don’t buy into the “running home to mummy and daddy” case - TFL were largely responsible for Anchor’s creation and it is the crown jewel in the ecosystem. It’s in everyone’s interest for it to succeed in a fair and responsible way.

As for the reserve, I think it should be reserved for being the reserve. It’s far too important.

I’m looking forward to hearing more about the details of the bug, chainlink and then forward.

[wengzilla]

I left two comments above about this:

TFL (or Anchor?) has been operating an extremely profitable liquidator (and I believe continues to place bids on Kujira)

Can we somehow use these profits that Anchor / TFL has earned through liquidations to fund the reimbursement? These liquidations were “never meant for profit” as per Do Kwon’s TG post below. Plus it would be back for TFL to profit off the back of Anchor (now that Anchor has to draw from ANC reserves to pay users back):

Another option is to consider using the $4.5B worth of Ozone reserves. These funds were meant to be used for “coverage of technical failure risks in the Terra DeFi ecosystem” (link to proposal), so using funds for this issue seems appropriate:

Alternatively, is it possible to use some of the UST allocated for Ozone reserves ($4B+) to pay for this? In my opinion, this is potentially the best option as it leaves the $ANC price intact, allows for a robust yield reserve, and would barely affect Ozone reserves.

[TomBlockUnify]

I think we should refund affected wallets with a max of $10000 refund, small/medium fish get off without any harm this way and the big accounts suck up and pay the debt for experimenting with DeFi.

[Anon_Luna]

I think it’s great that the issue is being resolved, congrats to Anchor.

Question is why should anyone be reimbursed? Good faith of the protocol! Good faith is communicating and resolving the issue.

My understanding is if you wish to participate in any thing that relies on a smart contract or Oracles, then you are taking risks, therefor it is your due diligence to limit your exposure to any risk by either buying insurance or not setting yourself up to be liquidated! Ie: reduce your LTV.

[geophf]

Hi. Thanks for posting this. I was wondering what Anchor’s response would be to the liquidations.

I was one of the 239ish people liquidated of both bETH and bLUNA. I understand you are taking a snapshot, but:

1. Is there any action you need from me to verify I was liquidated at 53% LTV?
2. Thank you for considering reimbursement, … is there a timeframe where I can know about being reimbursed?
3. Do you need from me the amounts of bLUNA and bETH I lost in this liquidation? Where do I communicate information about this incident?

I lost a lot (to me) in this liquidation, but mostly I lost confidence in the Anchor loan program. I’m glad Anchor is considering steps to rectify this incident.

[FreshLUNA]

Which oracle does anchor use now? Has this been implemented? Or does Anchor still use a separate oracle?

[ixus2021]

It would be good just give back user their liquidated coins and deduct the corresponding liquidated UST amounts from anchor. Mean reverting the liquidated transaction.

[charlietrumper]

Yes setting up a plan to sell and reimburse along a set time frame over x amount of days or weeks. Similar to how form 4 is filed with SEC for insider trading so it’s not all dumped at once. Obv no specific info given but dates which sales will happen would lessen the market drop.