AbstractPhila PRO

Using a type of planted value for utility, the process is yielding depth. Another 8k step arm will determine if this process can be made permanent, or if the 3s model is simply too deviated and requires an additional finetune pass.

This doesn't solve the FP instability of a fractal rotary, fibo rotary, standard rope, or a few others in conjunction with codebook trigrams; it just makes the problem harder for the optimizer to solve, leaving bread crumbs.

I'll figure something out. So far it's looking promising, but it's still falling quite short of standard models. With this format of dynamic modularization, there is a heavy burden to sharing the space.

I have a few ideas and I won't give up.

Report, present investigation, present evidence, move on. Sleep peacefully. That is the sort of job that I don't have time in my day to do, nor would I want to. The HF staff will decide what to do, as this is their platform to arbitrate. I'm personally having trouble keeping track of licenses for my models, but I still try to keep full dockets and logs of training data, systems used, formulas used and cited, but I'm likely still missing something here or there. Even with 10-20 different datasets used for a model, I still try to keep the citations and licenses together.

yeah but atp you have a model in your embedding space. so... its kinda a waste of compute yk.

How did you determine this is a waste of compute?

Show some decorum. Evidence can be presented reasonably and in understandable ways. Stronger cases can be shown through direct tool arbitration, testing 1:1 values with direct utilization, or one of a thousand other ways.

Use the scientific method to form a hypothesis, experiment, and conclusion. Repeat as necessary. None of this requires a game of text telephone in public.

I'm being cautiously optimistic. The in repair 2s softmax twin is keeping proper byte distance while the v3 arm to repair the distance is implemented and growing in skill.

This repair is working but it's touch and go, it may not fully solidify yet.

She is built in a modularity paradigm after all. Even with her byte recall distance being poor, she still keeps surprising me.

The new training methodology includes multiple methodologies for long form byte recall integration. Any used methodologies researched will be cited and attribution given.

V4 will be built entirely with long form byte recall, needle in a haystack, and multiple other test cases to ensure the model saturates her context window.

Alongside, the V4 model will likely be smaller, as a large variation is not required for the next test. V3 was too large to meaningfully train within a reasonable amount of time, while V2 can be trained considerably faster.

Looks like we've eliminated BF16 ATTENTION as the potential failpoint for the softmax as per 2 run checks, the model can continue to exist as BF16 attention most likely. Which is nice because it's very fast.

That being said something is happening in the BF16 weights causing a causal cascade that needs to be addressed.

FP16 backward projection from FP32 forward has stabilized the softmax version for version 2, and version 3 softmax version is in planning stages based on the results of 2s and the softmax structure upon training completion.

ETA a couple days.

https://colab.research.google.com/github/AbstractEyes/alephllm/blob/v0.10.9/notebooks/beatrix_2s_control_resume_colab.ipynb
2s variant's control form is ready to test. I have multiple benches to run.

The recall stability is an issue for v3. I've pinned down the causal factor for the measurements being inaccurate, and it's due to the BF16 memory being a large contributor to the cascade rounding errors. Even just converting the pretrained attention in the v3 model to fp32, the model regains some of her capability. It's just not enough to merit calling this model complete, not yet.

It's always something isn't it... I'll be renting a big group of cards to train a proper 2s control completion first, and then based on the outcome of the softmax on fp32 - if she destabilizes that is the cue for splat attention completely. If not, we're going to make v4 a hybrid, 4 layers splat per 1 layer softmax similar to how qwen operates.

The model's delta attention and memory may be getting rounded to death, and the viability of a solution to this isn't a very costly time sink, just a vram sink.

I'll be renting 16 48 gig cards to train the control variant of 2s's finished finetune in a day or two, and then depending on the outcome we'll be preparing the v3 control variant for 3 in a full fp32 attention spectrum.

Most of these arms that currently exist were prepared in a matter of a few days just on my single card, we have learned a great deal about what makes the model function and makes the model not function, and additionally we have learned a great deal about the potential of the tokenizers and their capacity within the model itself.

We will have many more answers with the completed 2s control, and if the result yields the v3 control will follow within less than a few days.

a

The tokenizer translation systems work just as expected, everything can simply exist as an arm.

https://huggingface.co/AbstractPhil/beatrix-tokenizers

One arm for translation for example Beatrix to Sentencepiece, another arm for a specific trained version of a sentencepiece model to generate similar and synthetic behavior to that model. The more data, the more similar Beatrix can represent the final hidden state of that model.

The token translation systems show recall at 99.5% accuracy at even the single direction overwhelmed fractal final layer, which means they work.

The similarity rankings vary from model to model, but they are considerably higher than placebo. With the quiet mechanism, Beatrix does not forget what she knows while she learns these arms. Bert ranking at >80% as I have the most possible Bert data compacted into the most useful way. The qwen from anima is on the list, as well as the T5, and multiple other models such as GPT2. A uniform qwen anima extraction with cc12m would take roughly 2.7 tb of data, so I'll need to operate at runtime which is slower, but we can't let a training cycle dominate the entire process with data movement.

With this, we have something that ought to procrustes rotate where she needs to rotate, and then we can flood her student with knowledge. Not just teach information, flood information.

Beatrix herself doesn't need to know the information, she just needs to be aware of what her own internal weights are similarly representing in comparison to what the expected outputs are meant to form. With that we rotate, whiten, and procrustes analyze the points. Suddenly, MSE and InfoNCE will provide exactly what the model needs to be an interpreter between two experts and a single student.

With that I'm updating the Abstract Powered Org to include updated information and support official releases, rather than just sitting there gathering dust.